On Fri, 15 Mar 2002, Mark Wisner wrote:
> I am working on a change to the IBM 405 Ethernet driver. The Ethernet
> allows us to define buffer sizes smaller than the MTU size. If a packet
> comes in larger than the buffer size, it will be received across multiple
> buffers. The current driver is a zero copy driver, it uses buffers of the
> MTU size, where we allocate a SKB for each descriptor and have the
> descriptor buffer pointer point to the SKB's data buffer. This allows the
> driver to avoid memory to memory moves.
> I would like to implement the same zero copy but I want to use smaller
> buffers. I would like to have each descriptor point to the buffer area of a
> separate SKB. When a large packet is received and it is received into
> multiple SKB's. I would like to pass these multiple SKBs to the stack. I
> want to avoid using memcopies. I think frag_list is what I want to use.
Presumably you are trying to receive into smaller chunks in an effort to
save memory. If so, your goal is misguided.
A significant part of the work in receiving packets is handling the
skbuffer allocation. By receiving large packets into multiple buffers,
you multiply that work. The overhead in setting up and following
multiple descriptors is also significant: descriptors might be much more
costly to access than normally cached memory since descriptors are
alternately accessed by the NIC and CPU.
Using multiple descriptors and buffers per packet will also make the PCI
bus utilization much worse. A modern PCI v2.1 NIC can write a received
Ethernet packet in a single linear burst. With multiple descriptors and
buffers the NIC must alternate between reading descriptors and writing
the data, re-arbitrating many times for a large packet.
Consider also that there is very little reduction in memory use when
using small buffers. Packets have bimodal size distribution correlated
with potential persistence: Packets are either small (e.g. acks) or full
sized (data). The small packets are quickly handled, while the full
sized packets are the ones that hang around waiting for the data to be
A final related note: drivers should always allocate consistently
sized skbuffs. Specifically, Ethernet drivers should always allocate
1536 byte buffers. Using a single buffer size gives the skbuff
allocator the opportunity to efficiently recycle buffers rather than
having to allocate and free a mix of sizes.
Donald Becker becker@xxxxxxxxx
Scyld Computing Corporation http://www.scyld.com
410 Severn Ave. Suite 210 Second Generation Beowulf Clusters
Annapolis MD 21403 410-990-9993