Mctrain's Blog

What I learned in IT, as well as thought about life

MSI-X How To

MSI-X How To Notes from linux-src/Documentation/PCI/MSI-HOWTO.txt


What are MSIs?

A Message Signaled Interrupt is a write from the device to a special address which causes an interrupt to be received by the CPU.

The MSI-X supports more interrupts per device than MSI and allows interrupts to be independently configured.

Why use MSIs?

3 reasons over pin-based interrupts:

  • pin-based PCI interrupts are often shared amongst several devices, false handler called when associated interrupt. While MSI are never shared.
  • in pin-based mechanism, interrupt may arrive before data has arrived in memory, so handler has to read a register on the specific device. MSI does not need this phase since the interrupt-generating write cannot pass the data writes
  • PCI device can only support one pin-based interrupt per function, so drivers has to query device what event occured. With MSI, a device can supoprt more interrupts for different purposes: e.g., give infrequent conditions their own interrupt, or give one interrupt to each packet queue in a network card or each port in a storage controller.

How to use MSIs?

1. Include kernel support for MSIs

CONFIG_PIC_MSI options enabled.

2. Using MSI

2.1 pci_enable_msi
int pci_enable_msi(struct pci_dev *dev)
  • allocate ONE interrupt to the device.
  • tevice switch from pin-based to MSI mode
  • dev->irq changed to a new numbre to repesent the MSI
  • this function should be called before request_irq(), because MSI is delivered via a vector different from vector of pin-based
2.2 pci_enable_msi_block
int pci_enable_msi_block(struct pci_dev *dev, int count)
  • this variation on the above call allows a device driver to request multiple MSIs. (power of 2, up to 25)
  • dev-irq to be the loweset of the new interrupts assigned to it.
2.3 pci_enable_msi_block_auto
int pci_enable_msi_block_auto(struct pci_dev *dev, unsigned int *count)
  • this variation on pci_enable_msi() call allows a device driver to request the maximum possible number of MSIs.
2.4 pci_disable_msi
void pci_disable_msi(struct pci_dev *dev)
  • restore dev->irq to the pin-based number, free MSIs
  • before this call, drivers should call free_irq() for each previously called request_irq().

3 Using MSI-X

It supports up to 2048 interrupts, each of which can be controlled independently.

To support this flexibility, drivers must use an array of struct msix_entry:

struct msix_entry {
  u16     vector; /* kernel uses to write alloc vector */
  u16     entry; /* driver uses to specify entry */
};
3.1 pci_enable_msix
int pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries, int nvec)
  • asks the PCI subsystem to allocate ‘nvec’ MSIs.
  • the ‘entries’ argument is a pointer to an array of msix_entry structs which should be at least ‘nvec’ entries in size.
  • on success, the device is switched into MSI-X mode and the function returns 0.
  • the ‘vector’ member in each entry is populated with the interrupt number;
  • the driver should then call request_irq() for each ‘vector’ that it decides to use.
  • the device driver is responsible for keeping track of the interrupts assigned to the MSI-X vectors to free later
  • this function does not adjust dev->irq. device will not generate interrupts for this dev-irq numbber once MSI-X is enabled.
3.2 pci_disable_msix
void pci_disable_msix(struct pci_dev *dev)
  • this function should be used to undo the effect of pci_enable_msix().
  • similar with pci_disable_msi
3.3 The MSI-X Table
  • the MSI-X capability specifies a BAR and offset within that BAR for the MSI-X Table.
  • this address is mapped by the PCI subsystem, and should not be accessed directly by the device driver.
  • if the driver wishes to mask or unmask an interrupt, it should call disable_irq() / enable_irq().

4. If a device implements both MSI and MSI-X capabilities, it can run in either MSI mode or MSI-X mode, but not both simultaneously.

5. Consideration when using MSIs

5.1 Choosing between MSI-X and MSI

MSI-X!

  • supports any number of interrupts between 1 and 2048, while MSI support only 32 (power of 2).
  • MSI interrupt vectors must be allocated consecutively, whiile MSI-X not
5.2 Spinlocks
  • With pin-based interrupts or a single MSI, it is not necessary to disable interrupts (Linux guarantees the same interrupt will not be re-entered).
  • for multiple interrupts, driver must disable interrupts while lock is held.
  • Two solutions:

    take the lock with spin_lock_irqsave() or spin_lock_irq() specify IRQF_DISABLED to request_irq() so that the kernel runs the entire interuupt routine with interrupts disabled.

6. How to tell whether MSI/MSI-X is enabled on a device

  • using “lspci -v”

MSI quirks

The PCI stack provides three ways to disable MSIs

  • globally
  • on all devices behind a specific bridge
  • on a single device

Comments