Overview / Usage

The goal of this project is to use existing python frameworks in order to come up with a general method for extracting data from retailers's websites, even when they may lack an api, and then use this data to accurately predict how quickly an item will sell. This application can be directly applied to real life problems in numerous ways. For example, a seller on ebay could easily use this algorithm to estimate how long it would take them to sell their item if they sold it at the average price, and also try to optimize the amount of time they’re willing to wait before selling, versus the amount of profit they make off the item. It also isn’t unreasonable to think that there is likely a direct correlation between how quickly something sells and how good of a deal it is. Now certainly this isn’t always true, but on average it seems like a fairly reasonable generalization. Thus, such a model could be used to predict potential “good deals”, and either alert the buyer or just purchase the items automatically (admittedly a much riskier, yet exciting line).

Methodology / Approach

I attempted to approach this problem of extracting data from ebay's website in the most general way possible. Although ebay does provide an api which can grab much of the requisite data I need, I used it as a last resort as I wanted to develop strategies for extracting data from numerous other websites regardless of the software interface they may provide. To speed up performance I took advantage of multithreading numerous times throughout the development process, in some cases speeding up my code by almost 10 times!

Technologies Used

BeautifulSoup, Selenium, Intel DevCloud, Ebay API

Other links

• Medium Post