I'm working with traffic data to try the reconstruct the origin-destination matrix of a region, based on the article Bell1983.
Basically, for each pair of locations, we need to get how many vehicles (/passengers) go from one place to the other, and we have data of public road traffic for it. This would be the O-D matrix. It's stored in the odm vector (flattened) with length M, the daily traffic for each road is stored in vector v with length N, and an extra matrix P, describing the probability of going through a certain road when going from place A to place B, stored in array P of size N x M (this matrix is constructed by some simple heuristic). In my case, there are 31 roads and 120 pairs of locations (16 choose 2).
The goal is the same as in the article: given constraints of the relation of these variables, get the solution that minimizes a functionf. In practicality, another important thing to note that constraint always contains singular matrix. (In the article they actually derive a general procedure for finding the solution, but for now I'm trying to write code to get a solution for any objective function.)
Models have the constraint v = P * odm so we have M equations for the M variables.
The problem is, in practical cases, P will be underdetermined because of things like 3 places laying on the same line (e.g. connected by 2 roads only, you have only 2 independent equations for 3 place pair values) thus P won't have an inverse. Same in my case, P has 28 independent equations only instead of 31. For this reason there are many possible solutions to the equation. Classical methods introduce some function and choose the solution minimizing this function - such a function is typically maximizing entropy (== minimizing (-1)*entropy).
Using Python, I've tried scipy.optimize.minimize with a simple function to minimize, given the constraint, but got the message: Singular matrix C in LSQ subproblem. I know, that P is singular and that caused this problem, but P will always be singular, underdetermined, I can only work with it. The code:
from scipy.optimize import minimize, Bounds#Objective functiondef f(odm): #Entropy maximizing (== minimizing the negative entropy) return np.sum(odm * np.log(odm))#Constraint(s)def constraint_eq(odm): return P @ odm - vconstraints = {'type': 'eq', 'fun': constraint_eq}bounds = Bounds(1e-5, np.inf) #positive values for logres = minimize(f, odm, constraints=constraints, bounds=bounds)optimal_odm = res.xThere will always be multiple solutions (and it is guaranteed that there will be at least one solution) and if the solution space is given it should be easy to find the one minimizing f.
How can I run an optimization that allows for singular matrices in constraint? Do I need to use a different solver method?
I know one possibility is to just exclude 3 equations to have a non-singular P matrix for optimization, then after having the optimal solution, solving the remaining equations, but this seems like quite some hussle.
(Solutions for MATLAB would also be useful, I'm familiar with it to some extent.)
For reproducing the problem, the road vector would be v = np.array([11479, 24663, 39783, 26064, 65054, 52134, 45172, 18807, 6386, 6544, 23218, 36905, 16558, 5385, 4562, 38232, 8263, 13132, 22395, 3759, 3910, 4100, 17482, 7736, 14255, 5154, 9436, 6554, 11623, 10747, 13110])
and the P-matrix stored as a CSV can be found here: GitHub.