Linux UIO for Microchip FPGA Designs

Do you want to quickly prototype a new FPGA device from Linux?

The Linux kernel provides a userspace I/O subsystem (UIO), originally written by Han J. Koch which enables you to write a simple driver almost entirely in userspace with only the very shell of the driver written in the kernel. The kernel uses a character device and sysfs to interact with a userspace process. Your user-space program can open, memory map the hardware's registers into userspace, and perform IO operations with. The userspace program can also use the device to check for interrupts.

This is a particularly useful technique if you are developing a custom peripheral on an FPGA such as Microchip's family as it is much faster to design the API to your hardware on Linux in user-space than in kernel space.

You can, of course, just use /dev/mem if you do not need interrupts. But, UIO gives you interrupts as well as memory.

Steps

  • Add a simple UIO driver to the Linux kernel
  • Build the Linux Kernel with support for UIO and your driver
  • Add a Device Tree Stanza describing your hardware
  • Implement a user-space driver

Adding a simple UIO driver to the kernel

Key Structures

There are two key kernel structures associated with the UIO subsystem.

The uio_info kernel structure associates a device name, with memory regions and irq handling.

struct uio_info
    - name: // device name
    - version: device driver version
    - irq: interrupt number or UIO_IRQ_CUSTOM
    - irq_flags: flags for request_irq() e.g. shared
    - handler: device's irq handler (optional)
        e.g. make sure that the interrupt has been generated by your hardware
        e.g. stop the interrupt
    - mem[]: memory regions that can be mapped to user-space

And the uio_mem structure describes a region of memory, in terms of start address, size and type.

struct uio_mem
    - addr: memory address
    - size: size of memory
    - memtype: type of memory region
        * UIO_MEM_PHYS
            - I/O and physical memory

Writing the Kernel Driver

I wrote my driver in drivers/uio/uio_mss.c

As usual, you need to associate the device tree stanza describing your hardware with this particular driver. The following two code snippets illustrate how to create a string that identifies this driver to the Device Tree subsystem and associates a probe function to be called early in Linux' boot to create the device driver for the hardware.

#if defined(CONFIG_OF)
static const struct of_device_id mss_dt_ids[] = {
    { .compatible = "microsemi,ms-pf-mss-uio"}
};
#endif

static struct platform_driver mss_driver = {
    .probe = mss_probe,
    .driver = {
        .name = DRV_NAME,
        .pm = MSS_PM_OPS,
        .of_match_table = of_match_ptr(mss_dt_ids),
        .owner = THIS_MODULE,
    };
};

module_platform_driver(mss_driver);

You'll need to implement the probe() routine. This code snippet illustrates the skeleton code you'll need to set up your hardware and register it with the UIO subsystem.

/* Implement a probe routine */
static int mss_probe(struct platform_device *pdev)
{
    dev_info(dev, "Running Probe\n");

    /* kzalloc memory for the driver itself */

    /* Generate a virtual address for the hardware's base addr */
    reg_base = platform_get_resource(pdev, IORESOURCE_MEM, 0);
    len = resource_size(reg_base);
    dev_info->reg_vaddr = ioremap(reg_base->start, len);

    /* Generate an IRQ entry for the hardware's IRQ */
    dev_info->irq = platform_get_irq(pdev, 0);

    /* Connect this info to the UIO subsystem */
    p->mem[0].addr = reg_base->start;
    p->mem[0].size = len;
    p->mem[0].memtype = UIO_MEM_PHYS;

    p->mem[1].size = 0; // sentinel

    p->name = kasprintf(GPF_KERNEL, "mss_uio0");
    p->version = "1.0";

    p->irq = dev_info->irq;
    p->irq_flags = IRQF_SHARED;
    p->handler = mss_uio_handler;  // We're going to provide our own handler

    // Disable interrupts, e.g.
    iowrite32(0, dev_info->reg_vaddr + INT_DIS_ENB_OFF);

    // Register this driver with UIO subsystem,
    uio_register_device(dev, p);

    dev_info(dev, "Created UIO device");
}

The probe routine states that this driver wants to handle interrupts itself, and this code snippet illustrates a basic interrupt handler.

static irqreturn_t mss_uio_handler(int irq, struct uio_info * info)
{
    struct uio_mss_dev * dev = info->priv;
    void __iomem * base = dev_info->reg_vaddr;
    void __iomem * int_enb = base + INT_DIS_ENB_OFFSET;
    void __iomem * int_stat = base + INT_STAT_OFFSET;

    status = ioread32(int_stat);
    enb = ioread32(int_enb);

    if( !(status & enb))
        return IRQ_NONE; // not for us

    iowrite32( (status & enb), int_stat);
    // uio subsystem looks after changing /dev/uio?
    // if user-space should handle 'losing' interrupts
    // or maybe extend kernel driver to record 'handled' interrupts.
    return IRQ_HANDLED;
}

Connect Your Kernel Driver to the Linux Config system

In drivers/uio, don't forget to associate your new kernel driver with your new config option.

So, in drivers/uio/Kconfig, add a new config UIO_MSS option, which depends on UIO.

Building your kernel

CONFIG_UIO=y
CONFIG_UIO_MSS=y

Device Tree

At a minimum, your device tree needs a node for your UIO device. I supplied a compatibility string to locate the correct driver, one memory area with one base address and one size, a clock, and an interrupt.

user_io@2000105000 {
    compatible = "microsemi,ms-pf-mss-uio";
    regs = <0x20 0x00105000 0x0 0x1000>;
    clock = <&xclk 0>;
    interrupt-parent = <&L4>;
    interrupts = <32>;
    status = "okay";
};

With the above device tree compiled and used by the kernel, you should see /dev/uio0 automatically created. You should not need to use mknod to create the device.

Userspace

A userspace program can use the UI device node as follows

  1. open() the device node (/dev/uio0) in read-write mode (O_RDWR)
  2. mmap() the hardware registers into userspace
  3. For example, enable hardware interrupts for the device.
  4. read() 32-bits from the device (/dev/uio0) to block until an interrupt arrives. You can also use select() or poll() or whatever blocking method you prefer. The read should take a pointer to a 32-bit value. That value will be updated with the total interrupt count for that IRQ line.
  5. disable the interrupts and close() the device to clean up.

API of UIO (in user-space)

There are two places in user-space where you can access your device driver. The first is in /sys/class/uio? which presents information about the device.

/sys/class/uio?/: information about device and UIO
    - name: UIO name
    - version: UIO version
    - maps/map?/: memory regions
        * addr: address of memory region
        * size: region size

The second place is in /dev/uio? which presents the hardware itself as a character device.

/dev/uio?: device acess
    - read(): wait for interrupts
    - mmap(): map device memory regions to user space
        * offset = region number * PAGESIZE

You can use the /sys/class interface to query properties of your hardware driver, for example, which /dev/uio? represents your device, how much memory you'll need to map, etc.

You can associate a user-friendly name with your device in your driver, e.g. "my-uio-driver". You can then traverse /sys/class/uio/uioX/name which associates /dev/uioX with "my-uio-driver".

For example, in python, you can use the following snippet to locate which /dev/uio? represents your hardware.

import os
if not os.listdir("/sys/class/uio"):
    ..
    sys.exit(0)

# For each /sys/class/uio/uio? directory
# find a file called 'name'
# if the contents of that file match the device name
# being searched for, use the corresponding uio? fragment
# to generate /dev/uio?.
for root, dirs, file in os.walk("/sys/class/uio"):
    for d in dirs:
        f = os.path.join(root, d)
        f = os.path.join(f, "name")
        dev = open(f, "r")
        dev = dev.read().replace('\n', '')
        if (dev == args.filename):
            dev = os.path.join("/dev/", d)
            # open dev

Example UIO user driver code

After locating your /dev/uio?, you can use the following code snippets to access your hardware, read and write the memory associated with your hardware and handle interrupts generated by the hardware in user space.

fd = open("/dev/uio0", O_RDWR | O_SYNC);

/* Map device's registers into user memory */
/* The mmap() size cannot exceed the memory size allocated by the kernel-space
   driver */
iomem = mmap(0, size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);

/* Enable interrupts */
iomem[INT_DIS_ENB_OFFSET] = 0xffff;

/* Do something */
iomem[TX_BUFFER] = 0x55555555;

/* Wait for an interrupt */
read(fd, &n_pending, sizeof(u_long));

/* Do Something */
rx = iomem[RX_BUFFER];

/* Tidy up */

/* Mark interrupts as dealt with */
iomem[INT_STATUS] = 0xffff;

/* Disable interrupts */
iomem[INT_DIS_ENB_OFFSET] = 0;

/* Unmap */
munmap(iomem, size);

/* Close */
close(fd);

Tips and Tricks

You can monitor /proc/interrupts to check the interrupt counts for the interrupt line associated with your hardware design. This lets you know if the interrupt is firing at all and/or whether an interrupt storm has occurred.

emdalo@emdalo:~/src/uio$ cat /proc/interrupts
           CPU1       CPU2       CPU3       CPU4
...
 47:     428927     435679     420410     457902  riscv,plic0,c000000  32  mss_uio0
...

Conclusion

  • UIO provides the ability to handle certain hardware designs in user-space

    • Low bandwidth designs;
    • Relatively latency tolerent;
    • Continuous physical memory allocation;
    • Interrupt handling.

  • UIO is useful for hardware development on FPGA

    • custom devices
    • exclusive use

Why not try this simple technique for getting your next hardware design up and running easily and quickly on Linux.

References