RaspberryPI Threading Causes Delay as time passes

Introduction

I'm developing a multi-threaded application using Python 3.9 on a Raspberry Pi, which utilizes ThreadPoolExecutor for managing multiple threads.

As time passes, there starts to have a delay in the execution of multiple functions. Initially, after a button is pressed, the function executes within a second, but after 2 days, the function takes close to 10 seconds before finishing its execution.

I have also noticed that the memory usage gradually increases over time, even though the queue length of tasks remains constant. I suspect it might be related to how threads are managed or how memory is handled within threads.

How the problem came about:

This happens only after I add the following functions:

def trigger_relay_one(thirdPartyOption=None):    outputPin = Relay_1    if thirdPartyOption == "GEN_OUT_1":        outputPin = GEN_OUT_1    if thirdPartyOption == "GEN_OUT_2":        outputPin = GEN_OUT_2    if thirdPartyOption == "GEN_OUT_3":        outputPin = GEN_OUT_3    try:        setGpioMode()        setupRelayPin(outputPin)        logger.info("Before toggleRelay1 thread")        thread_pool_executor.submit(toggleRelay1, outputPin, 'High', 5000, 1000, 1)        logger.info("After thread is submitted")    except RuntimeError:        returndef update_logs_and_server(dictionary):    def thread_task():        update(path +"/json/archivedLogs.json", archived_logs_lock, dictionary)        update(path +"/json/pendingLogs.json", pending_logs_lock, dictionary)        update_server_events()    thread_pool_executor.submit(thread_task)

The function above will be executed each time the RaspberryPi receives an input, for e.g. a press of a button. The above was implemented to prevent the updating of logs to be blocking, allowing for the RaspberryPi to continue to listen for inputs.

Tracemalloc and HTOP logs

The following are the logs as time passes:

Tracemalloc

// The following are three snapshots taken 24 hours apartTop 10 lines#1: /usr/lib/python3.9/threading.py:817: 2702.5 KiB    self._initialized = True#2: /usr/lib/python3.9/threading.py:381: 2315.4 KiB    self.notify(len(self._waiters))#3: /usr/lib/python3.9/threading.py:803: 1136.6 KiB    self._target = target#4: /usr/lib/python3.9/threading.py:522: 1066.9 KiB    self._cond = Condition(Lock())#5: /usr/lib/python3.9/threading.py:1205: 960.1 KiB    def invoke_excepthook(thread):#6: /usr/lib/python3.9/threading.py:820: 889.0 KiB    self._invoke_excepthook = _make_invoke_excepthook()#7: /usr/lib/python3.9/_weakrefset.py:85: 576.1 KiB    self.data.add(ref(item, self._remove))#8: /usr/lib/python3.9/threading.py:250: 441.2 KiB    self._waiters = _deque()#9: /usr/lib/python3.9/threading.py:231: 355.6 KiB    self._lock = lock#10: /usr/lib/python3.9/threading.py:930: 320.2 KiB    self._tstate_lock = _set_sentinel()813 other: 2401.7 KiBTotal allocated size: 13165.1 KiBTop 10 lines#1: /usr/lib/python3.9/threading.py:817: 12771.6 KiB    self._initialized = True#2: /usr/lib/python3.9/threading.py:381: 11052.4 KiB    self.notify(len(self._waiters))#3: /usr/lib/python3.9/threading.py:803: 5374.9 KiB    self._target = target#4: /usr/lib/python3.9/threading.py:522: 5041.5 KiB    self._cond = Condition(Lock())#5: /usr/lib/python3.9/threading.py:1205: 4537.3 KiB    def invoke_excepthook(thread):#6: /usr/lib/python3.9/threading.py:820: 4201.2 KiB    self._invoke_excepthook = _make_invoke_excepthook()#7: /usr/lib/python3.9/_weakrefset.py:85: 2536.5 KiB    self.data.add(ref(item, self._remove))#8: /usr/lib/python3.9/threading.py:250: 2031.8 KiB    self._waiters = _deque()#9: /usr/lib/python3.9/threading.py:231: 1680.5 KiB    self._lock = lock#10: /usr/lib/python3.9/threading.py:751: 1543.5 KiB    return template % _counter()793 other: 9620.2 KiBTotal allocated size: 60391.3 KiBTop 10 linesTop 10 lines#1: /usr/lib/python3.9/threading.py:817: 32927.9 KiB    self._initialized = True#2: /usr/lib/python3.9/threading.py:381: 28504.0 KiB    self.notify(len(self._waiters))#3: /usr/lib/python3.9/threading.py:803: 13854.6 KiB    self._target = target#4: /usr/lib/python3.9/threading.py:522: 12998.0 KiB    self._cond = Condition(Lock())#5: /usr/lib/python3.9/threading.py:1205: 11698.1 KiB    def invoke_excepthook(thread):#6: /usr/lib/python3.9/threading.py:820: 10831.5 KiB    self._invoke_excepthook = _make_invoke_excepthook()#7: /usr/lib/python3.9/_weakrefset.py:85: 5947.4 KiB    self.data.add(ref(item, self._remove))#8: /usr/lib/python3.9/threading.py:250: 5221.6 KiB    self._waiters = _deque()#9: /usr/lib/python3.9/threading.py:231: 4332.7 KiB    self._lock = lock#10: /usr/lib/python3.9/threading.py:751: 4007.5 KiB    return template % _counter()868 other: 23689.3 KiBTotal allocated size: 154012.5 KiB

Htop

// The following were snapshots taken 24 hours apart2024-04-17 14:59:28,126 - piProperty - INFO - 2024-04-17 14:59:28 - CPU Temp: 60.3°C, RAM Usage: 22.34%, CPU Usage: 28.80%14          2024-04-18 14:56:12,005 - piProperty - INFO - 2024-04-18 14:56:11 - CPU Temp: 62.3°C, RAM Usage: 33.13%, CPU Usage: 27.40%132024-04-19 14:55:45,997 - piProperty - INFO - 2024-04-19 14:55:45 - CPU Temp: 59.9°C, RAM Usage: 37.70%, CPU Usage: 25.70%142024-04-20 12:17:52,795 - piProperty - INFO - 2024-04-20 12:17:52 - CPU Temp: 61.3°C, RAM Usage: 41.46%, CPU Usage: 22.50%13                

Notes

I am quite certain that the increase in delay as time passes comes about with how the threading was implemented as the replacement of the thread_pool_executor.submit(toggleRelay1, outputPin, 'High', 5000, 1000, 1) to the blocking toggleRelay1(outputPin, 'High', 5000, 1000, 1) and similarly for thread_task does not result in any delay anymore as time passes.

Questions

1. Is the delay likely to be caused by the way threading was implemented?
2. Could the threading model in Python be causing memory leaks, even with apparent proper management and periodic garbage collection?
3. Are there better practices for managing memory and threads when using ThreadPoolExecutor in long-running applications?
4. Could there have been a better implementation instead of using a ThreadPoolExecutor if I want to make the updating of logs non-blocking?

If there are any more information required that may be helpful, I would greatly appreciate if you could drop me a comment, thank you so much!