Skip to main content

Path Dependence in Machine Learning: Looking Backward to Move Forward

While working on modeling the behavior of a component of the HVAC system using machine learning algorithms, I observed that when the data was changed to observe how the system would react, there were sharp movements in the model that didn't quite map out to reality. 


The issue bugged me. I knew there would be a solution. 

One day, while I was having my morning coffee and reading a paper on path dependence in economics, I saw the solution in front of my eyes: to perfectly capture the behavior of the system, not only do I have to feed the algorithm the present data, but the previous (t-1) data as well! 

Rushing to my laptop, I fixed the code to account for previous data as well as present data, and voila! It worked! I could easily replicate the behavior of the system with this very simple step.

Often times, data scientists work with data they are not familiar with, which leads to over-complications in the code/algorithm to reach the desired results, with little room for flexibility in the future.

But as engineers equipped with data science skills, simple solutions can unlock complex problems.