I need to read an RTSP stream, process the images individually in Python, and then write the images back to an RTSP stream. As the RTSP server, I am using Mediamtx [1]. For streaming, I am using FFmpeg [2].
I have the following code that works perfectly fine. For simplification purposes, I am streaming three generated images.
import timeimport numpy as npimport subprocesswidth, height = 640, 480fps = 25rtsp_server_address = f"rtsp://localhost:8554/mystream"ffmpeg_cmd = ["ffmpeg","-re","-f","rawvideo","-pix_fmt","rgb24","-s", f"{width}x{height}","-i","-","-r", str(fps),"-avoid_negative_ts","make_zero","-vcodec","libx264","-threads","4","-f","rtsp", rtsp_server_address,]colors = np.array( [ [255, 0, 0], [0, 255, 0], [0, 0, 255], ]).reshape(3, 1, 1, 3)images = (np.ones((3, width, height, 3)) * colors).astype(np.uint8)if __name__ == "__main__": process = subprocess.Popen(ffmpeg_cmd, stdin=subprocess.PIPE) start = time.time() exported = 0 while True: exported += 1 next_time = start + exported / fps now = time.time() if next_time > now: sleep_dur = next_time - now time.sleep(sleep_dur) image = images[exported % 3] image_bytes = image.tobytes() process.stdin.write(image_bytes) process.stdin.flush() process.stdin.close() process.wait()The issue is, that I need to run this at 10 fps because the processing step is heavy and can only afford 10 fps. Hence, as I reduce the frame rate from 25 to 10, the drop rate increases from 0% to 100%. And after a few iterations, I get a BrokenPipeError: [Errno 32] Broken pipe. Refer to the appendix for the complete log.
As an alternative, I can use OpenCV compiled from source with GStreamer [3], but I prefer using FFmpeg to make the shipping process simple. Since compiling OpenCV from source can be tedious and dependent on the system.
References
[1] Mediamtx (formerly rtsp-simple-server): https://github.com/bluenviron/mediamtx
[2] FFmpeg: https://github.com/FFmpeg/FFmpeg
[3] Compile OpenCV with GStreamer: https://github.com/bluenviron/mediamtx?tab=readme-ov-file#opencv
Appendix
Creating the source stream
To instantiate the unprocessed stream, I use the following command. This streams the content of my webcam as and RTSP stream.
ffmpeg -video_size 1280x720 -i /dev/video0 -avoid_negative_ts make_zero -vcodec libx264 -r 10 -f rtsp rtsp://localhost:8554/webcamError log
ffmpeg version 6.1.1 Copyright (c) 2000-2023 the FFmpeg developers built with gcc 12.3.0 (conda-forge gcc 12.3.0-5) configuration: --prefix=/home/conda/feedstock_root/build_artifacts/ffmpeg_1712656518955/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1712656518955/_build_env/bin/x86_64-conda-linux-gnu-cc --cxx=/home/conda/feedstock_root/build_artifacts/ffmpeg_1712656518955/_build_env/bin/x86_64-conda-linux-gnu-c++ --nm=/home/conda/feedstock_root/build_artifacts/ffmpeg_1712656518955/_build_env/bin/x86_64-conda-linux-gnu-nm --ar=/home/conda/feedstock_root/build_artifacts/ffmpeg_1712656518955/_build_env/bin/x86_64-conda-linux-gnu-ar --disable-doc --disable-openssl --enable-demuxer=dash --enable-hardcoded-tables --enable-libfreetype --enable-libharfbuzz --enable-libfontconfig --enable-libopenh264 --enable-libdav1d --enable-gnutls --enable-libmp3lame --enable-libvpx --enable-libass --enable-pthreads --enable-vaapi --enable-libopenvino --enable-gpl --enable-libx264 --enable-libx265 --enable-libaom --enable-libsvtav1 --enable-libxml2 --enable-pic --enable-shared --disable-static --enable-version3 --enable-zlib --enable-libopus --pkg-config=/home/conda/feedstock_root/build_artifacts/ffmpeg_1712656518955/_build_env/bin/pkg-config libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100Input #0, rawvideo, from 'fd:': Duration: N/A, start: 0.000000, bitrate: 184320 kb/s Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 640x480, 184320 kb/s, 25 tbr, 25 tbnStream mapping: Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))[libx264 @ 0x5e2ef8b01340] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2[libx264 @ 0x5e2ef8b01340] profile High 4:4:4 Predictive, level 2.2, 4:4:4, 8-bit[libx264 @ 0x5e2ef8b01340] 264 - core 164 r3095 baee400 - H.264/MPEG-4 AVC codec - Copyleft 2003-2022 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=4 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=10 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00Output #0, rtsp, to 'rtsp://localhost:8554/mystream': Metadata: encoder : Lavf60.16.100 Stream #0:0: Video: h264, yuv444p(tv, progressive), 640x480, q=2-31, 10 fps, 90k tbn Metadata: encoder : Lavc60.31.102 libx264 Side data: cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A[vost#0:0/libx264 @ 0x5e2ef8b01080] Error submitting a packet to the muxer: Broken pipe [out#0/rtsp @ 0x5e2ef8afd780] Error muxing a packet[out#0/rtsp @ 0x5e2ef8afd780] video:1kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknownframe= 1 fps=0.1 q=-1.0 Lsize=N/A time=00:00:04.70 bitrate=N/A dup=0 drop=70 speed=0.389x [libx264 @ 0x5e2ef8b01340] frame I:16 Avg QP: 6.00 size: 147[libx264 @ 0x5e2ef8b01340] frame P:17 Avg QP: 9.94 size: 101[libx264 @ 0x5e2ef8b01340] frame B:17 Avg QP: 9.94 size: 64[libx264 @ 0x5e2ef8b01340] consecutive B-frames: 50.0% 0.0% 42.0% 8.0%[libx264 @ 0x5e2ef8b01340] mb I I16..4: 81.3% 18.7% 0.0%[libx264 @ 0x5e2ef8b01340] mb P I16..4: 52.9% 0.0% 0.0% P16..4: 0.0% 0.0% 0.0% 0.0% 0.0% skip:47.1%[libx264 @ 0x5e2ef8b01340] mb B I16..4: 0.0% 5.9% 0.0% B16..8: 0.1% 0.0% 0.0% direct: 0.0% skip:94.0% L0:56.2% L1:43.8% BI: 0.0%[libx264 @ 0x5e2ef8b01340] 8x8 transform intra:15.4% inter:100.0%[libx264 @ 0x5e2ef8b01340] coded y,u,v intra: 0.0% 0.0% 0.0% inter: 0.0% 0.0% 0.0%[libx264 @ 0x5e2ef8b01340] i16 v,h,dc,p: 97% 0% 3% 0%[libx264 @ 0x5e2ef8b01340] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 0% 0% 100% 0% 0% 0% 0% 0% 0%[libx264 @ 0x5e2ef8b01340] Weighted P-Frames: Y:52.9% UV:52.9%[libx264 @ 0x5e2ef8b01340] ref P L0: 88.9% 0.0% 0.0% 11.1%[libx264 @ 0x5e2ef8b01340] kb/s:8.27Conversion failed!Traceback (most recent call last): File "/home/avishka/projects/read-process-stream/minimal-ffmpeg-error.py", line 58, in <module> process.stdin.write(image_bytes)BrokenPipeError: [Errno 32] Broken pipe