I have a simple FastAPI REST server running on AWS Lambda. It is writing logs to S3 using AWS Kinesis Firehose in batches of 20.
I have 2 problems related to background tasks:
- The write background tasks are blocking the main thread.
- Data loss if less than 20 events arrive, unless there is constant web traffic my write task does not finish and my periodic flush background task stops.
Only reliable option is to send 1 event at a time to Firehose, but that is slow. I tried a lot of workarounds, but I have run into problems with all of them.
FastAPI BackgroundTasks for batching events write
I use Boto3 lib to send events to Firehose, and that is not async. To keep latency low, I am putting events in an event queue, and writing batches of 20 events to Firehose, each batch is run in a FastAPI BackgroundTasks.
Maybe BackgroundTasks is not the right tool for AWS Lambda?
Blocking background tasks
To prevent waiting in my main thread, I do batch writes to Firehose in a normal function, not in an async. I can see that the batch write is running in a separate thread from the main thread. However nothing is happening in the main thread while this write is going on.
Background task shutdown when AWS Lambda is inactive
I have also tried to have an async periodic background task wake up once a minute and flush events that have not reached 20 for a batch write. However this background task shuts down unless there is constant traffic to the endpoint. This shutdown will also prevent a started background write task from finishing.
Increase Uvicorn worker count in AWS Lambda?
A possible problem is that the FastAPI's server, Uvicorn, by default is running with only 1 worker. I have also tried to find a place to change the number of Unicorn workers in Terraform or AWS console, but I have not found documentation on how to do this. I have even increased memory_size to 1.8 GB to get 2 cores for my microVM.
FastAPI on AWS Lambda or EC2
I cannot tell if my problem is a small misconfiguration or if it would be easier to run FastAPI on a static EC2 instance instead. On EC2 background tasks should work as expected, but integration with API Gateway is probably harder.