Linux Kernel Module that allows to map large PCIe DMA ring buffers.

It has been developed by cronologic GmbH & Co. KG to support the drivers for its time measurement devices such as:

• xHPTDC8 8-channel 13 ps streaming time-to-digital converter
• xTDC4 4-channel 13 ps common-start time-to-digital converter
• TimeTagger4 100/200/400/800 ps low cost TDCs
• Ndigo6G-12 6.4 Gsps 12 bit ADC digitizer board with pulse extraction and 13ps TDC.
• Ndigo5G-10 5 Gsps 10 bit ADC digitizer board with pulse extraction

However, the module could be useful to any PCIe developer who is using large buffers that are scattered in PCI space but shall look contiguous in user space.

The user mode part of the driver is available in a separate repository.

The initial code has been written by Bassem Ramzy with support from Richard Weinberger. It is licensed under GPL3.

Detailed instructions on how to install the project are given under Installation.

Here is just a brief overview of the necessary steps:

1. Install the prerequisites
2. Clone the repository using git clone https://github.com/cronologic-de/cronologic_linux_kernel.git
   Alternatively, download the packed source code (.zip or .tar.gz) of the latest release and unpack it
3. In a terminal, navigate to the repository folder and install the project using DKMS (see DKMS Installation)
4. The necessary libraries for your particular cronologic device are provided in its respective GitHub repository. To this end, follow the instructions within the respective repository on how to install the user guide example.
   • xHPTDC8
   • xTDC4
   • TimeTagger4
   • Ndigo6G-12
   • Ndigo5G-10
     

The project generates a Kernel Module that is installed on the system. This module creates a Miscellaneous Driver (miscdev) for every installed device, this miscdev driver is used to access its corresponding device.

Access is done by userspace and applications via the following ways:

1. sysfs. Accessing the driver from file system, e.g. accessing BAR0 or the Device Configuration.
2. ioctl calls for the miscdev file.

The driver is designed to be used for all cronologic devices that are installed on PCIe or Thunderbolt sockets.

dkms.conf is provided to install the driver module without the need to, manually, compile and build it, and supporting the automatic build and installation of the driver module with every new kernel version installed on the system.

The driver module in this project represents the Crono Kernel Layer in the architecture.

OS Abstraction Layer introduced in the Userspace Repository interfaces with the driver module via ioctl, mainly to manage DMA buffers.

.
├── include         # Header files to be included by userspace as well
├── src/            # Driver module source files
│   ├── debug_64    # Debug Build makefile and output files
│   └── release_64  # Release Build makefile and output files
├── Makefile        # Project general makefile 
└── install.sh      # Installation script

Driver is tested on Kernel Versions starting 5.0.

The driver is tested on the following 64-bit distributions:

(**) Driver code is built successfully, however, the driver was not tested on the devices.

make, gcc, and the Linux modules and headers need to be installed for your current kernel version. See Prerequisites for instructions on how to install them for the supported Linux distributions.

We recommend installing the project using DKMS, as outlined in DKMS Installation. If, however, you cannot use DKMS, installation instructions for a manual installation are also provided in Manual Installation.

If you only want to build the project and not install it, you can find instructions here.

sudo apt-get install make gcc

sudo apt-get install linux-headers-$(uname -r) linux-modules-$(uname -r)

sudo apt-get install dkms

sudo yum install gcc make

sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
sudo yum install https://www.elrepo.org/elrepo-release-9.el9.elrepo.noarch.rpm

sudo dnf --enablerepo=elrepo-kernel install kernel-lt
sudo dnf --enablerepo=elrepo-kernel install kernel-lt-devel

sudo dnf --enablerepo=elrepo-kernel install kernel-ml
sudo dnf --enablerepo=elrepo-kernel install kernel-ml-devel

sudo yum install dkms

sudo yum install gcc make

sudo yum install kernel-devel-$(uname -r)

sudo yum install dkms

sudo apt-get install make gcc

sudo apt-get install linux-headers-$(uname -r) linux-modules-$(uname -r)

sudo apt-get install dkms

sudo zypper install make gcc

sudo zypper in kernel-devel kernel-default-devel
sudo zypper up

• You can check if make and gcc are installed by running the following commands:

make -v

and

gcc -v

• You can check if kernel-devel is installed on your system by running the following command:

ls /lib/modules/$(uname -r)/build

Run dkms install command from the project folder, e.g.

sudo dkms install .

dkms.conf uses install.sh to install the project on the system and add module to the root.

The scripts support building the module in both compressed format .ko.zst and uncompressed format .ko.

Warning
If you manually install the project, you need to run the installation script with every kernel version used on the machine, and after every upgrade to a new kernel version. We recommend installation using DKMS.

This installation of the driver module is very simple, and is mainly done via insmod, however, an installation script is provided to support wider options, like debug, add to boot, uninstall, etc...

The installation firstly builds the driver module code, that's why the minimal build packages are needed as prerequisites (mentioned in the following sections).

To install the driver:

1. Clone this repository, either download it as .zip file and extract it, or use git.

git clone https://github.com/cronologic-de/cronologic_linux_kernel

2. From the Terminal, navigate into the directory cronologic_linux_kernel, for instance:

cd cronologic_linux_kernel

3. Run the installation script:

sudo bash ./install.sh

or, simply, run insmod

sudo insmod ../../../build/linux/bin/release_64/crono_pci_drvmod.ko

4. And, voilà, the driver module name is crono_pci_drvmod.

    sudo bash ./install.sh [Options]

    [Options]
    -s  (S)top currently loaded driver (if exists). if '-s 0', currently loaded driver
        will not be stopped, otherwise, (by default) it's stopped if loaded.
        Ignored if -i is not '0', or if -u is used.
    -i  (I)nstall driver. if '-i 0', driver will not be installed, otherwise,
        (by default) driver is installed. Ignored if -u is used.
    -b  Add driver to (B)oot. If '-b 0', driver will not be added to boot, otherwise,
        (by default) driver is added. Ignored if -u is used.
    -u  Uninstall the driver and remove it from boot startup.
    -d  Display (D)ebug Messages.
    -h  Display (H)elp and usage and exit.

• The script installs the release build of the driver, however, you can install the debug build of the driver, that is built automatically by the script, by replacing the line:

DRVR_INST_SRC_PATH="../build/linux/bin/debug_64/$DRVR_FILE_NAME.ko"

with

DRVR_INST_SRC_PATH="../build/linux/bin/release_64/$DRVR_FILE_NAME.ko"

The debug build of the driver module prints more information to the kernel messages displayed using dmesg.

• In case any error is encountered during installation, it should be either displayed explicitly on the terminal output or written in the error log file errlog found on the directory root.

• If the script encountered any error, it's highly recommended rerunning the script using -d (debug) option, which should provide further information about the step and command caused that error, e.g.

sudo bash ./install.sh -d

After installing the driver module successfully, the following takes place:

1. The driver module is listed in the system. You should find the module listed using lsmod, e.g.

$ lsmod | grep crono
crono_pci_drvmod       53248  0

2. The system probes all installed cronologic devices, and sends them to the driver, which, in turn, creates a Misc Driver for each device. You should find the misc drivers on the /dev directory:

ls /dev/crono* -lah

3. All driver messages are directed to the kernel messages displayed using dmesg

A typical debug successful output could be as following for two devices installed on the system (an xTDC4, and an xHPTDC8):

Run dkms remove command from the project folder, e.g.

sudo dkms remove crono_pci_drvmod/1.0.2 --all --force

It will both uninstall the driver module from dkms tree, and remove the module from the system and root by calling install.sh.

Sometimes the module file is not really deleted from the system, you might need to do the following to check it:

ls /lib/modules/$(uname -r)/kernel/drivers/pci/crono*

In case of major upgrade and the need to full cleanup, we might need to use the following command with the proper version:

sudo dkms remove crono_pci_drvmod/1.4.0 --all --force
sudo rm -rf /usr/src/crono_pci_drvmod-1.4.0 # Remove the source
sudo rm -rf build # On `cronologic_linux_kernel` folder
sudo dkms install . # Install again

Run the following command to uninstall the driver (if installed), and remove it from boot (if it's there):

sudo bash ./install.sh -u

Alternatively, you can run rmmod, which will only removes the installed module but won't remove it from boot if it's there.

sudo rmmod crono_pci_drvmod.ko

Note If you are using a cronologic device, you need to install the project instead of only building it. Please, refer to the Installation instructions.

Please refer to: Prerequisites

To build the project, run make command:

make

The build target output is:

• build folder is created on the same level of project folder cronologic_linux_kernel, to be shared with all other projects outputs in the same folder.

The following makefiles are used to build the project versions:

Run dkms build command from the project folder, e.g.

sudo dkms build .

dkms.conf uses the project Makefile found under /src/release_64 to build the project.

To clean the project all builds output:

make clean

Or, you can clean a specific build as following:

.../src/debug_64$ make clean
.../src/release_64$ make clean

For creating a kernel module, it's much simpler (and feasible) to have a Makefile per build, rather than having all builds in one Makefile.

Makefile for kernel module is simpler when having all the source code files in the same directory of the Makefile.

That's why, the Makefiles create symbolic links to the source files before starting the build, then delete them after building the code.

Userspace and applications can access the device via the following ways:

1. Using sysfs.
2. Using ioctl.

Full code is found in the userspace repository.

Accessing the miscdev via sysfs is mainly done via accessing files under directory /sys/bus/pci/devices/DDDD:BB:DD:F/.

Sample code:

        char dev_slink_path[PATH_MAX];
        char BAR0_resource_path[PATH_MAX];
        char dev_slink_content_path[PATH_MAX];
        ssize_t dev_slink_content_len = 0;
        int fd = -1;

        // Get the /sys/devices/ symbolic link content
        dev_slink_content_len =
            readlink("/sys/bus/pci/devices/0000:02:00.0", dev_slink_content_path, PATH_MAX - 1);
        if (-1 == dev_slink_content_len) {
                return -1;
        }
        // e.g. ../../../devices/pci0000:00/0000:00:1c.7/0000:02:00.0
        dev_slink_content_path[dev_slink_content_len] = '\0';

        // Build pPath
        sprintf(BAR0_resource_path, "/sys/%s/resource0", &(dev_slink_content_path[9]));

        // Open the BAR resource file
        fd = open(BAR0_resource_path, O_RDWR | O_SYNC);

        // Do mmap
        .
        .
        
        // Close the file
        close(fd);

        .
        .

Application should call ioctl using both the miscdev file name and the corresponding request value defined in ./include/crono_linux_kernel.h with prefix IOCTL_CRONO_.

Sample code to call ioctl:

#include "crono_linux_kernel.h"
{
        struct stat miscdev_stat;
        char miscdev_path[PATH_MAX];
        char miscdev_name[CRONO_DEV_NAME_MAX_SIZE];
        struct crono_dev_DBDF dbdf = {0, 0, 2, 0};
        CRONO_SG_BUFFER_INFO buff_info;

        // Get the miscdev file path, e.g. `/dev/crono_06_0002000`
        CRONO_CONSTRUCT_MISCDEV_NAME(miscdev_name, 0x06, dbdf);
        sprintf(miscdev_path, "/dev/%s", pDevice->miscdev_name);
        if (stat(miscdev_path, &miscdev_stat) != 0) {
                printf("Error: miscdev `%s` is not found.\n", miscdev_path);
                return -1;
        }

        // Open the miscdev
        miscdev_fd = open(miscdev_path, O_RDWR);
        if (miscdev_fd < 0) {
                printf("Error %d: cannot open device file <%s>...\n", errno,
                       miscdev_path);
                return -1;
        }

        // Initialize buffer information
        buff_info.xyz = abc ;

        // Call ioctl
        if (CRONO_SUCCESS !=
                ioctl(miscdev_fd, IOCTL_CRONO_LOCK_BUFFER, &buff_info)) {
                close(miscdev_fd);
                return -1 ;
        }

        // Close device file
        close(miscdev_fd);

        .
        .

}

• This example is provided for Scatter/Gather memory allocation, however, the driver provides functionality to lock contiguous memory directly as well using CRONO_CONTIG_BUFFER_INFO and IOCTL_CRONO_LOCK_CONTIG_BUFFER.

The misc driver name is constructed following the macro CRONO_CONSTRUCT_MISCDEV_NAME

crono_%02X_%02X%02X%02X%01X, device_id, domain, bus, dev, func

For example: the misc driver name is crono_06_0002000 for xTDC4 (Id = 0x06), domain = 0x00, bus = 0x02, device = 0x00, and function = 0x0.

To satisfy DMA APIs "guards", the driver code takes the following into consideration:

1. Calls pin_user_pages to pin the user pages, and calls unpin_user_pages when done working with it.
2. Allocates Scatter/Gather table using sg_alloc_table_from_pages, while its output is not really needed.
3. Uses dma_map_sg to map the Scatter/Gather table.

pin_user_pages is introduced, starting Kernel Version 5.6, to resolve problems caused by get_user_page as per this reference.

The driver uses pin_user_pages for kernel versions >= 5.6, and uses get_user_pages for kernel versions < 5.6.

Kernel version is determined in the Makefile, based on which, the identifier OLD_KERNEL_FOR_PIN is defined in case it's < 5.6.

A PC might have two devices of different types (models): e.g. xHPTDC8 and xTDC4. Each type is called a device type.

Miscellaneous devices are kept in crono_miscdev_pool.

The driver uses sg_alloc_table_from_pages as it optimizes memory (size of returned pages information) compared to __sg_alloc_table_from_pages, especially with huge memory sizes.

A macro is provided to toggle between the two functions, which is USE__sg_alloc_table_from_pages. By defining this macro, the code will use __sg_alloc_table_from_pages instead.

Use sg_alloc_table_from_pages to bind consecutive pages into one DMA descriptor to reduce size of S/G table sent to device.

Using __sg_alloc_table_from_pages instead of sg_alloc_table_from_pages will increase swiotlb buffer, when using dma_map_sg, it used 5868 (slots) to 32768 (slots), both ways, buffer is FULL:

crono_pci_driver 0000:05:00.0: swiotlb buffer is full (sz: 524288 bytes), total 32768 (slots), used 5868 (slots)

Using sg_dma_address is not applicable by “our driver” design when using sg_alloc_table_from_pages, while it is theoretically applicable when using __sg_alloc_table_from_pages and passing PAGE_SIZE.

When using sg_alloc_table_from_pages contiguous ranges of the pages are squashed into a single scatterlist entry up to the maximum size, hence, nents are not one-to-one mapped with the pages, so, we can’t really get use of sg_dma_address for page physical address.

As per Linux documentation, the number of pages returned by sg_alloc_table_from_pages is not necessarily equal to the number of input pages, actually in practice it’s much less, while the driver needs the physical address of every page of size 4096.

Since the driver uses sg_alloc_table_from_pages, accordingly, the driver uses PFN_PHYS(page_to_pfn()) to get the memory physical address.

BTW, I tried __sg_alloc_table_from_pages & sg_dma_address, but the addresses didn’t seem to be correct, but I didn’t use it again for the above-mentioned reason.

Kernel version is not preferred to be got using include <linux/version.h> and LINUX_VERSION_CODE identifier to cover that case when there are more than a kernel version installed on the environment.

For backward compability with kernel versions < 5.6, where pointers are not passed safely to ioctl, a new "integral unsigned" member variable (not a pointer, prefixed by u) is added to structures passed to ioctl for every pointer in the structure, this unsigned variable is set to the value of the address of the relevant pointer (by ioctl caller) and is used (instead of the pointer member variable) in the copy_from_user and copy_to_user.

For instance, the variable ucmds in CRONO_KERNEL_CMDS_INFO:

typedef struct {
        .
        CRONO_KERNEL_CMD *cmds;
        uint64_t ucmds; // Is used exchangeably with `cmds`.
                        // It is mainly provided for backward compatibility
                        // with kernel versions earlier than 5.6
        .
} CRONO_KERNEL_CMDS_INFO;

While, in kernel versions >= 5.6, it's safe to use the pointer member variables of the structures directly.

Moreover, if the structure member variable is a structure that has a pointer member, this "sub-pointer" needs to have a relevant u integral variable as well. That's why, we target to simplify all the structures passed to ioctl minimizing pointers as much as we can.

The following chart illustrates the relation between the file struct* (filp) that's passed to ioctl

int  ioctl(struct file *filp, ..);

with:

1. Its inode.
2. The calling process PID (userspace application).
3. The locked buffer.

Simple skeleton steps are like:

1. Register (e.g. using insmod) the kernel module, that registers a miscdev of the device, e.g. /dev/crono_06_0003000, of inodeA.
2. Run xtdc4_ug_example of Process ID 12345.
3. The process 12345 opens /dev/crono_06_0003000 by calling open(), returning file descriptor fdB.
4. The process 12345 calls ioctl() using fdB to lock BufferC (allocated previously in userspace 12345).
5. In the kernel module, ioctl() receives a struct file* of value filpD (in kernel space) that is related to fdB (in userspace).
6. The kernel module then locks the buffer BufferC and assign it to process 12345 (CRONO_SG_BUFFER_INFO_WRAPPER.app_pid).

Although a cronologic device can have one and only one ring buffer, but the chart was provided as a general example.

The source code files are formatted using clang-format, with LLVM format and IndentWidth: 8.

Sample dmesg output when successfully installing the driver, running the xTDC4 user guide example, then remove the module:

[  845.504198] crono: Registering PCI Driver...
[  845.504213] crono: Probe Device, ID <0x06>
[  845.504236] crono: Probed device BDBF: <0000:03:00.0>
[  845.504237] crono: Initializing cronologic miscdev driver: <crono_06_0003000>...
[  845.504349] crono: Done probing with minor: <116>
[  845.504367] crono: Done registering cronologic PCI driver
[  851.814986] crono: Done locking buffer: wrapper id <0>
[  851.815269] crono: Done locking buffer: wrapper id <1>
[  851.948633] crono: Done releasing buffer: wrapper id <0>
[  851.948793] crono: Done releasing buffer: wrapper id <1>
[  851.955006] crono: Done cleanup process PID<7858> buffer wrappers
[  860.159463] crono: Unregistering <1> miscellaneous devices...
[  860.159464] crono: Exiting cronologic miscdev driver: <crono_06_0003000>, minor: <116>...
[  860.159570] crono: Done exiting miscdev driver: <crono_06_0003000>
[  860.159572] crono: Done unregistering miscellaneous devices
[  860.159572] crono: Cleanup wrappers list...
[  860.159572] crono: Done cleanup wrappers list
[  860.159573] crono: Removing Driver...