application sleeps indefinitely trying to read gm_dma_malloc'ed buffer through O_DIRECT

["Ashutosh Rajekar" <arajekar@xxxxxxxxx>]

Hi,

We are running into a peculiar case of not being able to read files off of
an XFS filesystem anymore. This used to work earlier for us when we were
using a combination of linux kernel 2.4.27 (on RedHat 9.0) with GM 2.0.21.
We recently changed the distribution to Centos 4.1, and are using kernel
version 2.6.12.1.

The hardware platform is a Dell 2650, connected via a LSI Logic FC929 host
bus adaptor to an Adaptec Eurologic FC2502.

The application opens a GM port, and registers a few blocks of about 516K
memory using gm_dma_malloc(). The same buffers are used to write data to
and read from files residing on an XFS filesystem, created using
parameters
"/sbin/mkfs.xfs -f -i size=2048 -d agsize=4193792k -l
version=2,su=64k,size=128m /dev/sdc".

The application tries to write and read data using a 4K aligned offset
inside the 516K buffer, and does this using O_DIRECT.

Writing works fine, but reading a buffer back simple hangs that thread
process, inside a call to "filp->f_op->read( ...)".

Disabling the O_DIRECT flag makes things work fine. Using standard memory
allocated (i.e. memory not registered with GM using gm_dma_malloc()) using
a malloc() or a posix_memalign() makes sure it works fine too.

I am guessing this is an issue between gm_dma_malloc() pinning and locking
the memory for DMA transfers; and XFS not being able to handle memory page
locking/pinning when initiation a DIRECT I/O from that buffer, but only
when reading.

Any clues or suggestions?

Thanks and regards,
Ashutosh