Overview / Usage

Synopsis

Quantic is an analytics company using AI and machine learning models to provide decision support software to organizations across different industries.

It has developed a software that uses statistical methods to estimate the probability of error in an already-constructed predictive model. The user feeds that data and the outcome (predicted by the user’s model) into the Predictive Risk Analyzer and the analyzer's algorithm outputs information about the likelihood of the original prediction being off the mark. The user or analyst can use this tool to get a second opinion on how much business risk is being carried by his prediction owing to its inaccuracy.

This document describes briefly the real-life problem this software solves, and a peek into the solution’s design. It also includes a section on the bottlenecks seen during the development and how that was resolved.

Objective of the software

Organizations across industries support their business decisions through predictive models. They run their labelled data with appropriate machine learning algorithms to generate the trained model, and use the model to make a prediction about the target variable. The prediction from the model is validated by estimating the accuracy of the prediction using an unused, labelled sample data set.

Naturally, the user has a large variety of machine learning algorithms to choose from, and many options to statistically evaluate the accuracy of the model’s prediction.

That being said, if potentially significant business decisions will be taken based on the outcome predicted by the machine learning model, the user will benefit from an unbiased, statistically sound and rigorous re-validation of the prediction’s accuracy from an independent source.

That is what the Predictive Risk Analyzer tool from Quantic aims to go.

Methodology / Approach

Description of the Solution

This section explains the premise on which the Predictive Risk Analyzer is built, and gives a peek into the components under the hood of the solution. But first, a recap of the basics.

Recap of predictive model

The solution expects that the user has already created and used her predictive supervised learning model. It used two steps. In Stage 1, a predictive model was created using labelled data and an appropriate algorithm. In Stage 2, live data was fed into the trained model to get business prediction in production. It is this prediction that the user wishes to validate through the Prediction Risk Analyzer.

The solution’s premise

A predictive model is created by using a chosen machine learning algorithm on labelled historical data. Since an organization has only one set of relevant business data from the past, the data scientist can have just a single ‘run’ of the algorithm to train their model. They have the option of using a few candidate algorithms that fit the problem to run the models, but those trials again can be run only a few times.

Let us assume we have a coin to toss. We do not know if it is biased or not. If we had to make a prediction about heads or tails only after a handful of trial tosses, our confidence of correctly predicting the outcome of the next toss would be low, and potential of incorrect prediction high.

Working on the same principle, the Prediction Risk Analyzer approaches the solution by creating a statistically valid number of ‘tries’ for the prediction problem and measuring the outcome each time.

Each ‘try’ by the solution involves the following steps

1. Create statistically large set of synthetic, but realistic input data similar to real-life data originally used to forecast

2. Run the multiple models, using the synthetic data, and compute the predicted value of the target variable, and

3. Output the findings in quantitative as well graphical forms.

To illustrate our premise, let us take a common business scenario that may benefit from the Prediction Risk Analyzer.

The sales team of a consumer goods company RiskAverseCorp, estimates the next month’s expected sales figure as $ 5 million. This is predicted using its own model, and is based on applicable business inputs, or predictors. RiskAverseCorp will take some high-stakes business decisions, like ordering raw material, planning production, organizing inventory, etc. Before they fully commit to that plan, they wish to know how accurate the forecast is.

In other words, they want to know, what is the probability that the actual sales revenue matches with the value of $5 million predicted by the model. To enable this quantitatively, they can define a tolerance level in percentage terms. So long as the actual target value falls within that tolerance range from the predicted value, they consider it as ‘matched’.

Technologies Used

Software used

Application: The Prediction Risk Analyzer written by Quantic using Python

Machine learning / AI Library: The native sklearn library of Python

Environment: OneAPI cloud

Documents and Presentations

Repository

https://github.com/QuanticTechnovations/OneAPI_RiskAnalyzer