So overall, I'm sending 76,712 bytes over a socket in C++ and in my Python client, I'm seeing that my header that separates frames is showing up 65,536 bytes in. So here's what I've done.
In C++, I have a socket that I'm writing a 2d vector (array) to. I start with a header that defines the size of the array:
uint16_t num_vectors = static_cast<uint16_t>(vectors.size()); uint16_t vector_size = static_cast<uint16_t>(vectors[0].size()); std::cout << "Sending " << num_vectors << " vectors of size " << vector_size << " to client" << std::endl; std::ostringstream oss; oss << "header"; oss.write(reinterpret_cast<const char*>(&num_vectors), sizeof(uint16_t)); oss.write(reinterpret_cast<const char*>(&vector_size), sizeof(uint16_t)); std::string message = oss.str(); boost::asio::async_write(socket_, boost::asio::buffer(message), [](boost::system::error_code ec, std::size_t /*length*/) { if (!ec) { std::cout << "Vector information sent to client" << std::endl; } else { std::cerr << "Error sending vector information to client: " << ec.message() << std::endl; } });This all works. On the receiving side (python) I have:
header = await reader.readexactly(10) print(header) header_string, num_vectors, vector_size = struct.unpack('6sHH', header)and it outputs:
b'header\xac\x00\xbe\x01' b'header' 172 446That 172 x 446 is exactly what I expect.
Next back in the C++ code I send all the values of the vectors:
std::ostringstream oss2; for (const auto& vec : vectors) { for (uint8_t value : vec) { oss2.write(reinterpret_cast<const char*>(&value), sizeof(uint8_t)); } } std::string message2 = oss2.str(); std::cout << "Total message size: " << message2.size() << " bytes" << std::endl; boost::asio::async_write(socket_, boost::asio::buffer(message2), [](boost::system::error_code ec, std::size_t /*length*/) { if (!ec) { std::cout << "Vector information sent to client" << std::endl; } else { std::cerr << "Error sending vector information to client: " << ec.message() << std::endl; } });There should be (172 x 446) 76,712 bytes written as uint8_t are 1 byte. And I see that in my logs:
Total message size: 76712 bytesSo on the python side I have the ability to read in chunks (but I have also tried getting it all in one shot:
async def read_in_chunks(reader, expected_size, chunk_size=4096): data = bytearray() total_bytes_read = 0 while total_bytes_read < expected_size: chunk = await reader.readexactly(min(chunk_size, expected_size - total_bytes_read)) data.extend(chunk) total_bytes_read += len(chunk) return bytes(data)And I use that function to get the data:
expected_size = num_vectors * vector_size print(expected_size) data = await read_in_chunks(reader, expected_size) # Find the specified bytes within the data index = data.find(b'header\xac\x00\xbe\x01') if index != -1: print(index) vectors = [ list(data[i:i + vector_size]) for i in range(0, len(data), vector_size) ] vectors_np = np.array(vectors, dtype=np.uint8) print(vectors_np.shape)With output:
65536(172, 446)b'\x01\x01\xbe\x01\x01\x01\x01\x01\x01\x01'b'\x01\x01\xbe\x01\x01\x01' 257 25754360(257, 257)So it seems like the header is showing up 65,536 bytes into what I receive, then of course the rest goes off the rails because it's missing 11,176 bytes.