I have very basic knowledge in programming and I have been using some tips I found here, and for that thank you in advance.
Moving on to the problem, I am analyzing data from optical sensors that are measuring the extension caused to the pavement by vehicles.The simplest way I found to treat the raw file, since every 10 minutes 300.000 lines are generated in xlsx format, is a filtering of the noise with a kind of zscore algorithm. This algorithm will detect the most prominent peaks, but not all the ones I need. My Idea:
- Take the indices of these peaks that were detected and treat each one of them as a "block/passage" and after that, analyze each block individually.
- From the peak initially found, I wanted to define that I needed to collect the two seconds before and three seconds after, because this way the entire temporal space relative to each vehicle/passage would already be covered.
After the algorithm collected these 5 seconds, I needed:
- the count of peaks/axes of the vehicle
- the maximum values of each peak
- the increase in extension relative to the resting point (existing in the 2 seconds before the first peak).
I'm trying to dig in chatgpt but I always reach a dead end :/
Ill leave the algorithm im currently using for preliminary analysis below.Thank you so much
So Ive tried many approaches,including Savitzky filter to smooth the data and then find the peaks based on the derivative but the strategy that is giving me the best results is to calculate the moving average and standard deviation, and then define the peak based on that, adjusting the treshold, lag and influence of peaks.
import pandas as pdimport numpy as npimport matplotlib.pyplot as pltdf = pd.read_excel('2023_09_28_16_59_06.xlsx')lag = 500influence = 0.5threshold = 10def find_peak_indices(signal, lag, influence, threshold):peak_indices = [] peaks = [] processed_signal = [] processed_signal.append(signal[:lag])avg_list = []sd_list = []for index in range(lag, len(signal)):y = signal[index]avg_list.append(np.mean(processed_signal[-lag:]))sd_list.append(np.std(processed_signal[-lag:]))if abs(y - avg_list[-1]) > sd_list[-1] * threshold:peak_indices.append(index)peaks.append(signal[index]) if index > 0:adjusted_value = (influence * y) + ((1 - influence) * processed_signal[-1])else:adjusted_value = (influence * y) # No previous value, take y as isprocessed_signal = np.append(processed_signal, [adjusted_value])processed_signal = processed_signal[1:]return peak_indices, peakssignal = df[df.columns[16]]peak_indices, peak_values = find_peak_indices(signal, lag, influence, threshold)moving_avg = df[df.columns[16]].rolling(window=lag).mean()std_dev = df[df.columns[16]].rolling(window=lag).std()plt.figure(figsize=(12, 12)) plt.plot(signal, label='Signal')plt.plot(moving_avg, color='red', label='Moving Average')plt.title('Signal with Moving Average')plt.xlabel('Index')plt.ylabel('Value')plt.legend()plt.show()plt.plot(std_dev, color='green', label='Standard Deviation')plt.title('Standard Deviation')plt.xlabel('Index')plt.ylabel('Value')plt.legend()plt.show()plt.plot(signal, label='Signal')plt.scatter(peak_indices, peak_values, color='red', label='Peaks')plt.title('Signal with Peaks Detected') plt.xlabel('Index')plt.ylabel('Value')plt.legend()plt.show()print("Picos encontrados:")`for i in range(len(peak_indices)):``print("Índice:", peak_indices[i], "- Valor do pico:", peak_values[i])`