📄️ Creating Models
Models represent the various methods of machine learning provided by simMachines. simSearch is a similarity search engine, simRecommend is a collaborative recommendation engine, simClassify classifies and predicts, and simCluster identifies data clusters. Some models have variants, such as simClassify+, which is a Metric Learning classifier and simCluster+, which is a K-Means clustering model.
📄️ Finding Optimal Parameter Values for Classification
A simSearch model can be created without the need for extensive tuning. The advanced parameters let you change values that might affect training or query time. simClassify and simClassify+ do require tuning to get optimal results. The classification use-cases can vary considerably and will affect tuning.
📄️ Hyperparameter Tuning
To distinguish the Model Specifications or Parameters from data item specifications (or parameters), the Model Specifications are typically referred to as Hyperparameters. Finding the best ones to use is called Hyperparameter Tuning.
📄️ Grids
The usual way to find the best parameters is to create many different models with different combinations of parameters, and then examine the appropriate metrics. These experiments are referred to as Grid Experiments because it is driven by a grid of parameter combinations.
📄️ Auto Tune
Auto Tune requires the least amount of input from the user. It intelligently searches a large grid of experiments and only creates model experiments when the probability of successfully increasing the metric of interest is high. Sometimes it is able to find an optimal combination of hyperparameters by creating and evaluating only one third of all combinations. This efficiency will vary by dataset.
📄️ Exhaustive Grid Search
Exhaustive Grid Search will create a model and evaluate it for every possible combination in the grid. It will not skip any model experiment. Exhaustive Grid Search is done by turning off Auto Tune.
📄️ Grid Experiment Evaluation
ML Studio provides two ways to evaluate a model: Fold and Date Split. Which one to use depends upon the data set used for training.
📄️ Fold Experiment Mode
A fold evaluation does not depend upon the data types in the data set. So, in addition to being used for grid experiment evaluation, it can also be used to evaluate any model at any time. It is present in the Model Actions menu of a classification model, for example:
📄️ Date Split Experiment Mode
If the dataset you are using to train the model has a date column and if it makes sense to divide up the dataset by that date for training and testing, then you can use the date split evaluation.
📄️ Interpreting Grids and Grid Results
When you click on the Grid Results item in the Model Action menu, you are taken to the Grid Results page, which has two tabs. The first tab is the Grid Table and has one row per grid experiment. You can download the results of a specific grid experiment and remove it from the table.
📄️ Grid Analysis Report: ROC Curve
A useful metric for determining the appropriate classifier for an application is AUC Receiver Operating Characteristics.)
📄️ Interpreting Multinomial Fold Experiments
The results of a multinomial fold experiment are shown in a table, similar to grid results. Model metrics are displayed in the table.
📄️ Domain Optimization for Classification
Classification using simClassify+ has extra parameters and features for optimizing results based upon a domain attribute. The domain attribute can be any REAL column and could represent money, effort, size, and so on. Typically, this would be used in fraud detection with the domain attribute being the amount of money in a transaction. In this example, this directs the metric learner to optimize not just for fraud transactions caught, but fraud money caught.
📄️ Weighted Recall
Weighted recall is a new metric to be used in grid result analysis. Based on a REAL column that the user selects from the training data set in the grid creation process, this metric will evaluate the percentage of that chosen metric caught for each class (e.g. % of fraud dollars caught). This is identical in logic to the standard recall metrics, but focuses on the percentage of a REAL variable caught by the model, instead of the percentage of the number of records caught.
📄️ Thresholding
Thresholding, a feature used in simClassify and simClassify+ predictors, acts as a limit to split resulting confidence values into a true or false category for binomial predictions. Thresholds only apply to confidence levels for the positive class of a prediction (i.e. the class with the lower number of predictions). If the confidence value for the positive class exceeds the threshold value, the result is considered true and the positive class is predicted. If the confidence value is less than the threshold, the result is false and the negative class is predicted.
📄️ Model Calibration
Use the Model Calibration feature to recalibrate a model's prediction probabilities to reflect a different class distribution than that of the training distribution. Model Calibration trains and deploys an Isotonic Regression model to adjust prediction probabilities of a simClassify+ model. Model Calibration can be used in combination with Downsampling to reduce training time on highly class imbalanced data sets.