I want to replace backlashes in python, replacing \\ by \ in a byte sequence

So I have a byte sequence that looks like this:

key = b"b')F\\x87\\x86\\xfc~Z\\xaa\\x15pf}\\xe6\\x9f\\xf5\\xd0'"

and I want it to look like this: b')F\x87\x86\xfc~Z\xaa\x15pf}\xe6\x9f\xf5\xd0'

I am manipulating the sequence in bytes and not converting it in string because I need it the sequence to remain in byte type, when I decode() it and encode() it again the format changes and using the .replace method is pointless so working with the replace method on the byte sequence is what I have found to be most practical...except for the backlashes.

I use the replace method to manipulate the byte sequence so it looks the way i want it with the following code:

modified_bytes = key.replace(b"b'", b"").replace(b"'", b"").replace(b'', b'').replace(b'\\\\',b'\\')

When I print it, it gives meb')F\\x87\\x86\\xfc~Z\\xaa\\x15pf}\\xe6\\x9f\\xf5\\xd0'

which is almost the result I am looking for..Except for the backlashes, I am stuck and I cannot get rid of them even with the .replace(b'\\\\',b'\\') , is it even possible?