I'm working on a Python project where I need to filter a large list of dictionaries based on multiple conditions.
Here's a simplified example of the data structure I'm dealing with:
data = [ {"name": "David", "age": 30, "city": "Nairobi"}, {"name": "Shanice", "age": 25, "city": "Kisumu"}, {"name": "Shon", "age": 35, "city": "Mombasa"}, # ... potentially thousands of more entries]I want to filter this list to include only those entries where the age is greater than 28 and the city is either "Nairobi" or "Mombasa". My current approach is to use a list comprehension like this:
filtered_data = [ entry for entry in data where entry['age'] > 28 and entry['city'] in ('Nairobi', 'Mombasa')]While this works for smaller lists, performance becomes an issue with larger datasets. Is there a more efficient way to perform this filtering, especially for very large lists?
Any suggestions or best practices for optimizing this kind of operation would be greatly appreciated!