Quantcast
Channel: Active questions tagged python - Stack Overflow
Viewing all articles
Browse latest Browse all 23305

Losing Data writing to socket between C++ and python?

$
0
0

So overall, I'm sending 76,712 bytes over a socket in C++ and in my Python client, I'm seeing that my header that separates frames is showing up 65,536 bytes in. So here's what I've done.

In C++, I have a socket that I'm writing a 2d vector (array) to. I start with a header that defines the size of the array:

    uint16_t num_vectors = static_cast<uint16_t>(vectors.size());    uint16_t vector_size = static_cast<uint16_t>(vectors[0].size());    std::cout << "Sending " << num_vectors << " vectors of size " << vector_size << " to client" << std::endl;    std::ostringstream oss;    oss << "header";    oss.write(reinterpret_cast<const char*>(&num_vectors), sizeof(uint16_t));    oss.write(reinterpret_cast<const char*>(&vector_size), sizeof(uint16_t));    std::string message = oss.str();    boost::asio::async_write(socket_, boost::asio::buffer(message),        [](boost::system::error_code ec, std::size_t /*length*/) {            if (!ec) {                std::cout << "Vector information sent to client" << std::endl;            } else {                std::cerr << "Error sending vector information to client: " << ec.message() << std::endl;            }        });

This all works. On the receiving side (python) I have:

    header = await reader.readexactly(10)    print(header)    header_string, num_vectors, vector_size = struct.unpack('6sHH', header)

and it outputs:

    b'header\xac\x00\xbe\x01'    b'header' 172 446

That 172 x 446 is exactly what I expect.

Next back in the C++ code I send all the values of the vectors:

    std::ostringstream oss2;    for (const auto& vec : vectors) {        for (uint8_t value : vec) {            oss2.write(reinterpret_cast<const char*>(&value), sizeof(uint8_t));        }    }    std::string message2 = oss2.str();    std::cout << "Total message size: " << message2.size() << " bytes" << std::endl;    boost::asio::async_write(socket_, boost::asio::buffer(message2),        [](boost::system::error_code ec, std::size_t /*length*/) {            if (!ec) {                std::cout << "Vector information sent to client" << std::endl;            } else {                std::cerr << "Error sending vector information to client: " << ec.message() << std::endl;            }        });

There should be (172 x 446) 76,712 bytes written as uint8_t are 1 byte. And I see that in my logs:

    Total message size: 76712 bytes

So on the python side I have the ability to read in chunks (but I have also tried getting it all in one shot:

async def read_in_chunks(reader, expected_size, chunk_size=4096):    data = bytearray()    total_bytes_read = 0    while total_bytes_read < expected_size:        chunk = await reader.readexactly(min(chunk_size, expected_size - total_bytes_read))        data.extend(chunk)        total_bytes_read += len(chunk)    return bytes(data)

And I use that function to get the data:

    expected_size = num_vectors * vector_size    print(expected_size)    data = await read_in_chunks(reader, expected_size)    # Find the specified bytes within the data    index = data.find(b'header\xac\x00\xbe\x01')    if index != -1:        print(index)    vectors = [        list(data[i:i + vector_size]) for i in range(0, len(data), vector_size)    ]    vectors_np = np.array(vectors, dtype=np.uint8)    print(vectors_np.shape)

With output:

65536(172, 446)b'\x01\x01\xbe\x01\x01\x01\x01\x01\x01\x01'b'\x01\x01\xbe\x01\x01\x01' 257 25754360(257, 257)

So it seems like the header is showing up 65,536 bytes into what I receive, then of course the rest goes off the rails because it's missing 11,176 bytes.


Viewing all articles
Browse latest Browse all 23305

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>