coro_rpc, from yalantinglibs, is a high-performance RPC library based on C++20 coroutines. It provides a concise and easy-to-use interface, allowing users to implement RPC communication with just a few lines of code. In addition to TCP, coro_rpc now also supports RDMA communication (ibverbs). Let's use a simple example to get a feel for coro_rpc with RDMA communication.

example ​

Start the RPC server:

std::string_view echo(std::string str) { return str; }
coro_rpc_server server(/*thread_number*/ std::thread::hardware_concurrency(), /*port*/ 9000);
server.register_handler<echo>();
server.init_ibv(); // Initialize RDMA resources
server.start();

The client sends an RPC request:

Lazy<void> async_request() {
  coro_rpc_client client{};
  client.init_ibv(); // Initialize RDMA resources
  co_await client.connect("127.0.0.1:9000");
  auto result = co_await client.call<echo>("hello rdma");
  assert(result.value() == "hello rdma");
}

int main() {
  syncAwait(async_request());
}

With just a few lines of code, you can set up an RPC server and client based on RDMA communication. If users need to configure more RDMA-related parameters, they can pass a config object when calling init_ibv(). Various ibverbs-related parameters can be set in this object. For details, see the documentation.

How do you enable TCP communication? Simply don't call init_ibv(). TCP is the default communication protocol; RDMA communication is enabled only after init_ibv() is called.

benchmark ​

We conducted some performance tests on coro_rpc between two hosts in a 180Gb RDMA (RoCE v2) environment. In high-concurrency, small-packet scenarios, the QPS can reach 1.5 million. When sending slightly larger packets (256KB and above), the bandwidth can be easily saturated with fewer than 10 concurrent connections.

The specific benchmark code can be found here.

Automatic GID Selection ​

In RDMA communication, a GID (Global Identifier) is a global address used to identify a device port. A single RDMA device port may have multiple GID entries (different protocol types, different IP addresses), and the optimal GID must be selected to establish a connection.

coro_rpc implements automatic GID selection logic internally, so users do not need to manually specify a gid_index. The selection rules are as follows:

Filtering Rules ​

• For RoCE-type GIDs, entries with ndev_ifindex == 0 (i.e., no associated network device) are excluded
• IB-type GIDs are not subject to this restriction (the IB protocol does not depend on the IP network stack)

Priority Ordering ​

Device type priority (high → low):

1. IB (InfiniBand)
2. RoCE v2 (based on UDP/IP, routable)
3. RoCE v1 (based on Ethernet, not routable)

Address priority within the same type (high → low):

1. Global unicast address: Routable IPv6 or IPv4-mapped global addresses (e.g., fd03::, 2001::, ::ffff:10.x.x.x)
2. Link-local address: fe80::/10 or ::ffff:169.254.x.x, valid only on the local link
3. Loopback address: ::1 or ::ffff:127.x.x.x

Example ​

Using a real GID table from an mlx5 bond device:

The automatic selection result is INDEX 3 (RoCE v2 + global unicast address), which is the optimal choice.

GPU-direct RDMA Support ​

GPU-direct RDMA allows direct memory access between GPU memory and remote nodes via RDMA, eliminating the dependency on CPU during data transfers. This feature significantly reduces latency and improves throughput for GPU-related applications.

Initialization ​

To enable GPU-direct RDMA support, you need to:

1. Initialize CUDA Environment: First get available CUDA devices and initialize the GPU environment:

   cpp

   auto cuda_dev_list = coro_io::cuda_device_t::get_cuda_devices();

2. Create IB Device with GPU Memory Support: Create an InfiniBand device that supports GPU memory buffers:

   cpp

   auto dev = coro_io::ib_device_t::create(
       {.buffer_pool_config = {.gpu_id = 0 /* GPU ID */}});

3. Initialize Server and Client with GPU Buffer Support IB Device:

   • Server: server.init_ibv({.device = dev})
   • Client: client.init_ibv({.device = dev})

RPC Client ​

• Set Request Attachment: Use set_req_attachment2 to send GPU data:

  cpp

  coro_io::data_view gpu_attachment; // = ...;
  client.set_req_attachment2(gpu_attachment);

• Access Response Attachment: Use get_resp_attachment2 to retrieve GPU data sent by the server from the response:

  cpp

  coro_io::data_view resp_attachment = client.get_resp_attachment2();

• Optional: Set Response Attachment Buffer: Use set_resp_attachment_buf2 to pre-allocate and set the buffer address for receiving attachment response data. When the length is insufficient, an internal buffer will be automatically reallocated.

  cpp

  coro_io::data_view gpu_attachment_buf; // = ...;
  client.set_resp_attachment2(gpu_attachment_buf);

  data_view is a data view that, in addition to traditional [data()](file:///root/lizezheng/yalantinglibs/include/ylt/thirdparty/asio/detail/is_buffer_sequence.hpp#L38-L38) and [size()](file:///root/lizezheng/yalantinglibs/include/ylt/thirdparty/asio/detail/is_buffer_sequence.hpp#L35-L35) interfaces, provides a [gpu_id()](file:///root/lizezheng/yalantinglibs/include/ylt/coro_io/memory_owner.hpp#L66-L72) interface to indicate the GPU ID where the GPU memory resides. When ID=-1, it indicates that the data is located in system memory.

RPC Server ​

On the RPC server side, you can access request attachments and set response attachments through the context of the RPC function:

// In the handler function
void rpc_function() {
    coro_io::data_view attachment = coro_rpc::get_context()->get_request_attachment2();
    coro_rpc::get_context()->set_response_attachment2(attachment);
}

Performance Advantages ​

GPU-direct RDMA eliminates CPU-GPU memory copying during network transmission, reducing latency and CPU overhead. Data flows directly from GPU memory to the network interface and vice versa, making it ideal for high-performance computing and AI applications where large amounts of GPU data need to be shared across nodes.

GPU CRC32 Async Computation (via nvCOMP) ​

coro_io::cuda_crc32_async asynchronously computes the CRC32 of a single device buffer on a GPU stream; the caller reads the result after stream.record().get(). It is backed by nvcompBatchedCRC32Async from NVIDIA nvCOMP and defaults to the CRC-32/PKZIP polynomial — pass nvcompCRC32_C (iSCSI), nvcompCRC32_BZIP2, or any other preset to switch algorithms.

Enabling (CMake options) ​

This feature requires nvCOMP v4.0 or newer. Install it on the system first:

# RHEL / Alibaba Cloud Linux
sudo yum install -y nvcomp-cuda-12   # or nvcomp-cuda-11 / nvcomp-cuda-13

# Ubuntu / Debian
sudo apt-get install -y nvcomp-cuda-12

Then enable both YLT_ENABLE_CUDA and YLT_ENABLE_NVCOMP at CMake configure time, and explicitly specify the nvCOMP include / library paths (no auto-search, because the install location differs across CUDA major versions):

cmake -B build \
  -DYLT_ENABLE_CUDA=ON \
  -DYLT_ENABLE_NVCOMP=ON \
  -DNVCOMP_INCLUDE_DIR=/usr/include/nvcomp_12 \
  -DNVCOMP_LIB=/usr/lib64/libnvcomp.so

Required options:

If YLT_ENABLE_NVCOMP=ON but NVCOMP_INCLUDE_DIR or NVCOMP_LIB is not set, CMake fails with a FATAL_ERROR showing the expected usage.

Usage ​

#include <ylt/coro_io/cuda/cuda_crc32.hpp>

coro_io::cuda_stream_handler_t stream{0};  // gpu_id must be explicit
auto d_data = coro_io::cuda_malloc_async(stream, len);   // device input
auto d_crc  = (uint32_t*)coro_io::cuda_malloc_async(stream, sizeof(uint32_t));

coro_io::cuda_crc32_async(stream, (void*)d_data, len, d_crc);              // default PKZIP
// coro_io::cuda_crc32_async(stream, (void*)d_data, len, d_crc, nvcompCRC32_C);  // or iSCSI

YLT_CHECK_CUDA_ERR(stream.record().get());  // sync before reading *d_crc

Calling conventions (same as cuda_copy_async):

• device_data and device_out must reside in device memory
• The CRC32 result is not yet available when the function returns; the stream must be synchronized before reading *device_out
• Default polynomial is CRC-32/PKZIP; pass any nvCOMP preset constant to switch algorithms

RDMA Performance Optimization ​

RDMA Memory Pool ​

RDMA requests require pre-registering memory for sending and receiving data. In our tests, the overhead of registering RDMA memory is much higher than that of memory copying. A more reasonable approach, compared to registering memory for each send or receive operation, is to use a memory pool to cache already-registered RDMA memory. Each time a request is initiated, data is divided into multiple chunks for receiving/sending. The maximum length of each chunk corresponds to the size of the pre-registered memory blocks. A registered block is retrieved from the pool, and a memory copy is performed between this block and the actual data address.

RNR and the Receive Buffer Queue ​

RDMA operates directly on remote memory. If the remote memory is not ready, it triggers an RNR (Receiver Not Ready) error. To handle an RNR error, one must either disconnect or sleep for a period. Clearly, avoiding RNR errors is key to improving RDMA transfer performance and stability.

coro_rpc uses the following strategy to address the RNR issue: For each connection, we prepare a receive buffer queue. This queue contains several memory blocks (e.g., 8 blocks of 256KB by default). Whenever a notification of a completed data transfer is received, a new memory block is immediately added to the buffer queue, and this new block is posted to RDMA's receive queue.

Send Buffer Queue ​

In the send path, the most straightforward approach is to first copy data to an RDMA buffer, then post it to the RDMA send queue. After the data is written to the peer, repeat the process for the next block.

This process has two bottlenecks: first, how to parallelize memory copying and network transmission; second, the NIC is idle during the time between when it finishes sending one block and when the CPU posts the next, failing to maximize bandwidth utilization.

To improve sending performance, we introduce the concept of a send buffer. For each I/O operation, instead of waiting for the peer to complete the write, we complete the send operation immediately after posting the memory to the RDMA send queue. This allows the upper-level code to send the next request/data block until the number of outstanding sends reaches the send buffer queue's limit. Only then do we wait for a send request to complete before posting a new memory block to the RDMA send queue.

For large packets, this algorithm allows memory copying and network transmission to occur concurrently. Since multiple blocks are sent simultaneously, the NIC can send another pending block during the time it takes for the application layer to post a new one after the previous one is sent, thus maximizing bandwidth utilization.

Small Packet Write Coalescing ​

RDMA throughput is relatively low when sending small packets. For small packet requests, a strategy that improves throughput without introducing extra latency is to coalesce multiple small packets into a larger one. If the application layer posts a send request when the send queue is full, the data is not immediately sent but is temporarily stored in a buffer. If the application then posts another request, we can merge the new data into the same buffer as the previous data, achieving data coalescing for sending.

Inline Data ​

Some RDMA NICs can send small packets using the inline data method, which does not require registering RDMA memory and offers better transfer performance. coro_rpc uses this method for packets smaller than 256 bytes if the NIC supports inline data.

Memory Usage Control ​

RDMA communication requires managing memory buffers manually. Currently, coro_rpc uses a default memory chunk size of 256KB. The initial size of the receive buffer is 8, and the send buffer limit is 2. Therefore, the memory usage per connection is 10 * 256KB, approximately 2.5MB. Users can control memory usage by adjusting the buffer queue sizes and the buffer block size.

Additionally, the RDMA memory pool also provides a watermark configuration to control the upper limit of memory usage. When the memory pool's watermark is high, a connection attempting to get a new memory block from the pool will fail and be closed.

Using a Connection Pool ​

In high-concurrency scenarios, the connection pool provided by coro_rpc can be used to reuse connections, avoiding the overhead of repeated connection creation. Furthermore, since coro_rpc supports connection multiplexing, we can submit multiple small packet requests over a single connection. This enables pipelined sending and leverages the underlying small packet write coalescing to improve throughput.

auto pool = coro_io::client_pool<coro_rpc::coro_rpc_client>::create(
    conf.url, pool_conf);
auto ret = co_await pool->send_request(
    [&](coro_io::client_reuse_hint, coro_rpc::coro_rpc_client& client) {
        return client.send_request<echo>("hello");
    });
if (ret.has_value()) {
    auto result = co_await std::move(ret.value());
    if (result.has_value()) {
        assert(result.value()=="hello"); 
    }
}