Federated Averaging somehow always gives identical accuracy on test set

I am implementing federated averaging on the IoT-23 dataset(lighter version)

Omitting the preprocessing I am saving the data into a test set and 9 train sets as below:

X_train, X_test, Y_train, Y_test = train_test_split(X, y, random_state=10, test_size=0.2)test_set = pd.concat([X_test, Y_test], axis=1)test_set.to_csv('./dataset/test_set.csv')num_sets = 9set_size = len(X_train) // num_setstrain_sets = np.array_split(train_set, num_sets)for i in range(num_sets):    train_sets[i].to_csv(f'./dataset/train{i}.csv')#type:ignore

My model looks like this:

model = models.Sequential([        layers.Input(shape=(24,)),        layers.Dense(150,activation='relu'),        layers.Dense(80,activation='relu'),        layers.Dropout(0.2),        layers.Dense(7, activation='softmax')    ])    loss_fn = losses.CategoricalFocalCrossentropy(alpha=0.2)    model.compile(loss=loss_fn, optimizer='rmsprop', metrics=['accuracy'])

Now with these I am training and saving weights for each of the train sets:

for client_number in range(9):    model = models.load_model('./model/model.keras')    train_data=pd.read_csv(f"./dataset/train{client_number}.csv",dtype='float')    X_train = train_data.iloc[:, 1:-7]    y_train = train_data.iloc[:, -7:]    base_model.fit(X_train, y_train, epochs=5)    model.save_weights(f'./weights/weight{client_number}.weights.h5')#type:ignore

As you can see from the screenshot each of them does give different accuracies while training

Now for federated averaging I am using this sum_scaled_weights() method but without adding weights since I've divided the dataset into equal sizes.

def sum_weights(weight_list):    averaged_weights = list()    #get the average grad accross all client gradients    for grad_list_tuple in zip(*weight_list):        layer_mean = tf.math.reduce_sum(grad_list_tuple, axis=0)        averaged_weights.append(layer_mean)    return averaged_weights

I am aggregating the weights with something like:

model=models.load_model('./model/model.keras')for i in range(3):    model.load_weights(f'./weights/weight{i}.weights.h5')#type:ignore    org_weights.append(model.get_weights())#type:ignoreaverage_weights = sum_weights(org_weights)model.set_weights(average_weights)loss, accuracy = model.evaluate(X, y)#type:ignoreprint(f'Org Aggregate no.{j} accuracy: {accuracy}')

Now the issue I'm running into is that regardless of how many weights I aggregate, the accuracy when checking for test ends up the exact same.eg. with 3 of them seperately (i.e weights 0-2,3-5,6-8):

with all 9:

I have tried the test set with the initial weights instead and they do give me different accuracies:

I'm honestly stumped and curious about how and why this is happening. Any explanation or corrections for any error in the logic would be appreciated.