Overview / Usage

Kotekan is a high performance data processing pipeline for radio telescopes. Currently it is used by the Canadian Hydrogen Intensity Mapping Experiment (CHIME) Radio Telescope as the X-engine and real time beamformer. It is also used at the Algonquin Radio Observatory (ARO) as a data capture backend, and for radio astronomy labs at the University of Toronto with software defined radio (SDR) receivers. In the case of CHIME it handles over 6.4 Tb/s of radiometric data in real time, and performs computations at a rate of almost 10 TeraOps, using a cluster of 256 servers containing a total of 1024 GPUs. This is achieved using off-the-shelf computer hardware: x86 Intel CPUs, AMD GPUs, Intel based network interface cards (NICs). The framework is designed to be extensible for use with other next generation telescopes. Kotekan is supported by a large team of academics and developers working across many universities and institutes.

Methodology / Approach

Kotekan is designed to be framework for creating radiometric data processing pipelines. The framework uses processing "stages" (system threads or groups of system threads), joined by "buffers" (currently supporting ring buffers, with planned support for other queuing models). Stages and buffers are linked to create a data pipeline based on a custom YAML file format. This allows for different pipelines to be constructed by simply creating new configuration files.

Radio correlators are often large and expensive, but fortunately tend to have deterministic data rates and processing requirements, thus significant cost savings can be had by achieving high throughput per compute node. As a result, the framework is focused on making efficient use of CPU, GPU, and networking resources. Our approach to this includes minimizing the use of expensive high level APIs and languages, and where possible reducing load heavy operations like memory copies and memory allocation while in operation. Other low level optimizations include using huge pages, pinned memory, zero-copy operations, carefully tuned GPU kernel code, and AVX intrinsics in critical sections.

Technologies Used

This project is written in C++14, with some python. For high-speed networking it uses the Data Plane Development Kit (DPDK) along with Intel based network cards (e.g. XL710), and socket IO for lower data rate networking. To process the data it uses Intel CPUs combined with GPUs using the OpenCL API, and the AMD ROCm API, with future support for Nvidia's CUDA API planned. For archiving data it primarily uses the HDF5 file format. Other libraries used include libevent, OpenBLAS, FFTW, and airspy. The software runs on Linux (Ubuntu and RHEL/CentOS) and MacOS (with limited support for some subsystems), and x86 64-bit CPUs. YAML is used for configuration files. The application is interacted with using a REST style HTTP interface, and statistics are automatically exported for use by the Prometheus monitoring software.

Repository

https://github.com/kotekan/kotekan

Other links

• Project Documentation