On Fri, 1 Jul 2005, Alan Cox wrote:
> > No it's not. You need to insert appropriate barriers, one of: wmb(),
> > mb() or rmb(). In rare cases you may need to use iob(), which is
> > currently non-portable (which reminds me I should really push it
> > upstream).
> Its even more complicated than that 8)
> writeb/writel may be merged in some cases (but not re-ordered) for I/O
Is that non-reordering specified anywhere for the API or does it just
happen to be satisfied by most implementations? Ours (for MIPS, that is)
for example does nothing to ensure that.
> devices but a simple mb() will only synchronize them as viewed from
> cpu/memory interface. There are two other synchronization points. From
That's true -- which is why I mentioned bridge-specific operations may be
> the bridge with the I/O device (typically the PCI root bridge) which is
> not enforced automatically across processors on some large numa boxes
> but is not usually a problem and on the PCI bus itself.
What if the host I/O bus is not PCI? For this kind of stuff I tend to
think in the terms of TURBOchannel systems, just to be sure not to get
influenced by the most common hardware. ;-)
E.g. I have this R4400-based TURBOchannel system with aggressive
buffering in the CPU's MB (memory buffer) ASIC which requires a read-back
(RAM is OK for that) after a write and a memory barrier only to make
writes propagate to the I/O bridge. It may be worse yet with TURBOchannel
Alpha and VAX systems. With the latters TURBOchannel is behind two
bridges, with two intermediate buses on the way.
> PCI permits posting (delaying writes) and some forms of merging (but not
> re-ordering). Thus if you need an I/O to hit a device on the PCI bus and
> know it arrived you must follow it by a read from the same device. So
> for example if you want to shut down a DMA transfer and free the buffer
> for a PCI device you
> need to do
> writel(TURN_DMA_OFF, dev->control);
> /* Only now is the free safe */
Again, the I/O bus your host is attached to need not be PCI and you may
need a bridge specific operation to make your write be completed, possibly
combined with your quoted sequence (if there is actually PCI somewhere in
the system; think AlphaServer 8400).