Classification Example

Next, we will look at an example of how to call a classification model to score data using the RASON modeling language. To open this example, click RASON Example Models on the Editor tab of www.RASON.com, Data Science – Data Mine Classification – Fitted Models POSTed to Server – NeuralNetwork.json.

This example imports two datasources (hald-small-binary-train.txt and hald-small-binary-valid.txt) and binds both to a dataset (myTrainData and myValidData, respectively). A fitted model is created, nncEstimator, which is then fit to the myTrainData dataset. This model, nncModel, is used to score both datasets and produce statistics on how well the model fits each dataset.

 {
    "modelName": "NeuralNetworkClassification",
    "modelType": "datamining",
    "modelDescription": "classification: neural network",
    "datasources": {
        "myTrainSrc": {
            "type": "csv",
            "connection": "hald-small-binary-train.txt",
            "direction": "import"
        },
        "myValidSrc": {
            "type": "csv",
            "connection": "hald-small-binary-valid.txt",
            "direction": "import"
        }
    },
    "datasets": {
        "myTrainData": {
            "binding": "myTrainSrc",
            "targetCol": "Y"
        },
        "myValidData": {
            "binding": "myValidSrc",
            "targetCol": "Y"
        }
    },
    "estimator": {
        "nncEstimator": {
            "type": "classification",
            "algorithm": "neuralNetwork",
            "parameters": {
                "priorProbMethod": "EMPIRICAL",
                "numNeurons": [4],
                "numEpochs": 100,
                "errorTolerance": 0.01,
                "weightDecay": 0.0,
                "learningRate": 0.4,
                "weightMomentum": 0.7,
                "hiddenLayerActivation": "LOGISTIC_SIGMOID",
                "outputLayerActivation": "SOFTMAX",
                "weightInitSeed": 123,
                "learningOrder": "RANDOM",
                "learningOrderSeed": 1234,
                "dataForErrorComputation": "TRAIN_AND_VALID",
                "maxNumEpochsWithNoImprovement": 30,
                "minRelativeErrorChange": 1E-05,
                "maxTrainingTimeSeconds": 5
            }
        }
    },
    "actions": {
        "nncModel": {
            "trainData": "myTrainData",
            "validData": "myValidData",
            "estimator": "nncEstimator",
            "export": "json",
            "action": "fit",
            "evaluations": [
              "trainingLog", 
              "neuronWeights", 
              "numEpochsUsed", 
              "trainingTime", 
              "stoppingReason", 
              "partitionCausedStopping"
            ]
        },
        "trainScore": {
            "data": "myTrainData",
            "fittedModel": "nncModel",
            "action": "predict",
            "parameters": {
                "successClass": "1",
                "successProbability": 0.6
            },
            "evaluations": [
              "prediction", 
              "posteriorProbability", 
              "confusionMatrix", 
              "accuracy", 
              "specificity", 
              "sensitivity", 
              "recall", 
              "precision", 
              "f1"
            ]
        },
        "validScore": {
            "data": "myValidData",
            "fittedModel": "nncModel",
            "action": "predict",
            "parameters": {
                "successClass": "1",
                "successProbability": 0.6
            },
            "evaluations": [
              "prediction", 
              "posteriorProbability", 
              "confusionMatrix", 
              "accuracy", 
              "specificity", 
              "sensitivity", 
              "recall", 
              "precision", 
              "f1"
            ]
        }
    }
}
    

In the example above, two datasets are imported in "datasources". The first dataset, hald-small-binary-train.txt, is used to train the Neural Network while hald-small-binary-valid.txt is used to evaluate, or "validate" the fitted model. The first dataset, hald-small-binary-train.txt is imported as myTrainSrc and the second dataset, hald-small-binary-valid.txt, is imported as myValidSrc.

Note: RASON allows unused data sources to be defined. In this example, two data sources are exported, myPMMLSrc and myJSONSrc, but only one datasource, myJSONSrc, is actually used during the fit (See binding: myJSONSrc within dataSources). However, the "fit" action can point to at most 1 export binding. As a result, the file that is bound to "fit" will be populated while the unused file will be empty.

Within "datasets", "myTrainSrc" is bound to "myTrainData" and "myValidSrc" is bound to "myValidData". Note: Input files in a Data Science Rason model must not contain a path to a file location.

Within "estimator", the estimator "nncEstimator" is created using the neural network classification algorithm ( "type": "classification", "algorithm": "neuralNetwork" ). Various options are set for this algorithm under "parameters". For a complete list of the options available to the neural network classification algorithm, see the Rason reference guide.

Three action items are created within "actions", nncModel, trainScore and validScore.

The nncModel action item, fits the neural network classification model, using the estimator nncEstimator, to the training dataset, "myTrainData". The fitted model will be run on the dataset, myValidData, in order to evaluate the performance of the model. The fitted model is assigned the name given to the modelName setting, i.e. modelName: 'NueralNetworkPostFM'. The fitted model will be run on the dataset, myValidData, in order to evaluate the performance of the model. The following results are requested within "evaluations": the training log (trainingLog), the neuron weights (neuronWeights), number of epochs (numEpochsUsed), time to train the model (trainingTime), reason for stopping (stoppingReason) and the partition that caused the algorithm to stop (partitionCausedStopping).

The trainScore action item, scores the myTrainData dataset, using the fitted model nncModel, using a Success Class equal to 1 and a success probability equal to 0.6. The following results are requested for the training partition scoring action item: predictions (prediction), the posterior probabilities (posteriorProbability), the confusion matrix (confusionMatrix) and the 6 statistics (accuracy, specificity, sensitivity, recall, precision and f1).

Lastly, the validScore action item scores the myValidData dataset using the fitted model nncModel. The following results are requested for the validation partition scoring action item: predictions (prediction), the posterior probabilities (posteriorProbability), the confusion matrix (confusionMatrix) and the 6 statistics (accuracy, specificity, sensitivity, recall, precision and f1).

Here are the results from this Rason model. Notice the results begin with the status code = 0, id = "2590+NeuralNetworkClassification+2020-01-20-01-55-36-971921" and codeText = "success".

Getting model results: GET https://rason.net/api/model/2590+NeuralNetworkClassification+2020-01-20-01-55-36-971921/result
{ 
 "status": {
  "id": "2590+NeuralNetworkClassification+2020-01-20-01-55-36-971921",
  "code":0,
  "codeText":"Success"
 },

This is the start of the nncModel action results. Notice the estimator parameters are printed.

  "results":""results": ["nncModel.trainingLog", "nncModel.neuronWeights", "nncModel.numEpochsUsed", "nncModel.trainingTime", 
  "nncModel.stoppingReason", "nncModel.partitionCausedStopping", "trainScore.prediction", "trainScore.posteriorProbability", 
  "trainScore.confusionMatrix", "trainscore.accuracy", "trainScore.specificity", "trainScore.sensitivity", "trainScore.recall", 
  "trainScore.precision", "trainScore.f1", "validScore.prediction", "validScore.posteriorProbability", "validScore.confusionMatrix", 
  "validScore.accuracy", "validScore.specificity", "validScore.sensitivity", "validScore.recall", "validScore.precision", "validScore.f1"],

Under "nncModel" – "actions" – "evaluations", in the Rason model above, we asked for the following results: 'trainingLog', 'neuronWeights', 'numEpochsUsed', 'trainingTime', 'stoppingReason' and 'partitionCausedStopping'.

The trainingLog results, as requested, for nncModel contains 100 rows and 4 columns.

Row names are found under "rowNames", column headings under "colNames" and data is found under "data".

"nncModel":{ 
  "trainingLog": {
     "objectType": "dataFrame",
     "name": "Training Log",
     "order": "col",
     "rowNames": [
        "Epoch 1", "Epoch 2", "Epoch 3", "Epoch 4", "Epoch 5", "Epoch 6",
        "Epoch 7", "Epoch 8", "Epoch 9", "Epoch 10", "Epoch 11", "Epoch 12", "Epoch
        13", "Epoch 14", "Epoch 15", "Epoch 16", "Epoch 17", "Epoch 18", "Epoch 19",
        "Epoch 20", "Epoch 21", "Epoch 22", "Epoch 23", "Epoch 24", "Epoch 25",
        "Epoch 26", "Epoch 27", "Epoch 28", "Epoch 29", "Epoch 30", "Epoch 31",
        "Epoch 32", "Epoch 33", "Epoch 34", "Epoch 35", "Epoch 36", "Epoch 37",
        "Epoch 38", "Epoch 39", "Epoch 40", "Epoch 41", "Epoch 42", "Epoch 43",
        "Epoch 44", "Epoch 45", "Epoch 46", "Epoch 47", "Epoch 48", "Epoch 49",
        "Epoch 50", "Epoch 51", "Epoch 52", "Epoch 53", "Epoch 54", "Epoch 55",
        "Epoch 56", "Epoch 57", "Epoch 58", "Epoch 59", "Epoch 60", "Epoch 61",
        "Epoch 62", "Epoch 63", "Epoch 64", "Epoch 65", "Epoch 66", "Epoch 67",
        "Epoch 68", "Epoch 69", "Epoch 70", "Epoch 71", "Epoch 72", "Epoch 73",
        "Epoch 74", "Epoch 75", "Epoch 76", "Epoch 77", "Epoch 78", "Epoch 79",
        "Epoch 80", "Epoch 81", "Epoch 82", "Epoch 83", "Epoch 84", "Epoch 85",
        "Epoch 86", "Epoch 87", "Epoch 88", "Epoch 89", "Epoch 90", "Epoch 91",
        "Epoch 92", "Epoch 93", "Epoch 94", "Epoch 95", "Epoch 96", "Epoch 97",
        "Epoch 98", "Epoch 99", "Epoch 100"
     ],
     "colNames": [
        "Training: Network Error (Cross Entropy)", "Training: Data Error
       (Misclassification)", "Validation: Network Error (Cross Entropy)",
       "Validation: Data Error (Misclassification)"
     ],
     "colTypes": ["double", "double", "double", "double"],
     "indexCols":"null",
     "data": [
        [0.66303106458569439, 0.61046150383331155, 0.56944006636282729,
         0.52779162039300587, 0.48417504066980516, 0.44362260680789289,
         0.40638475616914688, 0.37215396361664815, 0.3411592111360856,
         0.31431372121567258, 0.29131909802834266, 0.27102389766093798,
         0.25353556252847897, 0.23813822487069872, 0.22470097813635537,
         0.21280936293863378, 0.20228703681028415, 0.19291095351064058,
         0.1845022151930211, 0.1769364291783426, 0.17008566901396463,
         0.16385202086310643, 0.15815736165219668, 0.1529302492035593,
         0.14811227303762814, 0.14365570720572687, 0.13951863318194754,
         0.13566514472547134, 0.13206511877311614, 0.12869251061694048,
         0.12552454521962747, 0.1225415301884923, 0.11972637230850892,
         0.11706403575912855, 0.11454124699820405, 0.11214629679558265,
         0.10986881018953637, 0.10769953657381337, 0.10563020106791242,
         0.1036533877314311, 0.10176242944089263, 0.099951309541604375,
         0.098214582156025687, 0.096547306563537702, 0.09494498943028741,
         0.093403533610519077, 0.091919194140355126, 0.090488540652579968,
         0.089108424598987587, 0.087775950263900768, 0.086488449223837263,
         0.085243457964565922, 0.084038698196981468, 0.082872059416664465,
         0.08174158339000176, 0.080645450347748282, 0.07958196668651682,
         0.078549553981234477, 0.077546739135733622, 0.076572145533259556,
         0.075624485073849626, 0.074702550998768227, 0.073805211411634764,
         0.072931403416608628, 0.072080127805153577, 0.071250444232266827,
         0.070441466830221316, 0.069652360213760117, 0.068882335836058628,
         0.068130648659690554, 0.067396594111116953, 0.066679505290812818,
         0.065978750414207185, 0.065293730461298868, 0.064623877015205694,
         0.063968650272027958, 0.063327537206275833, 0.062700049877748992,
         0.062085723867207059, 0.061484116829451943, 0.06089480715359024,
         0.060317392721253721, 0.059751489754459056, 0.059196731745587997,
         0.058652768462681568, 0.058119265023883551, 0.057595901035437588,
         0.057082369788157505, 0.056578377507750292, 0.056083642654785618,
         0.055597895270477579, 0.055120876364782037, 0.054652337343614694,
         0.05419203947226759, 0.053739753372352525, 0.053295258549820014,
         0.052858342951806907, 0.052428802550251424, 0.052006440950377902,
         0.051591069022308984],
        [0.15384615384615385, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0.66303106458569439, 0.61046150383331155, 0.56944006636282729,
         0.52779162039300587, 0.48417504066980516, 0.44362260680789289,
         0.40638475616914688, 0.37215396361664815, 0.3411592111360856,
         0.31431372121567258, 0.29131909802834266, 0.27102389766093798,
         0.25353556252847897, 0.23813822487069872, 0.22470097813635537,
         0.21280936293863378, 0.20228703681028415, 0.19291095351064058,
         0.1845022151930211, 0.1769364291783426, 0.17008566901396463,
         0.16385202086310643, 0.15815736165219668, 0.1529302492035593,
         0.14811227303762814, 0.14365570720572687, 0.13951863318194754,
         0.13566514472547134, 0.13206511877311614, 0.12869251061694048,
         0.12552454521962747, 0.1225415301884923, 0.11972637230850892,
         0.11706403575912855, 0.11454124699820405, 0.11214629679558265,
         0.10986881018953637, 0.10769953657381337, 0.10563020106791242,
         0.1036533877314311, 0.10176242944089263, 0.099951309541604375,
         0.098214582156025687, 0.096547306563537702, 0.09494498943028741,
         0.093403533610519077, 0.091919194140355126, 0.090488540652579968,
         0.089108424598987587, 0.087775950263900768, 0.086488449223837263,
         0.085243457964565922, 0.084038698196981468, 0.082872059416664465,
         0.08174158339000176, 0.080645450347748282, 0.07958196668651682,
         0.078549553981234477, 0.077546739135733622, 0.076572145533259556,
         0.075624485073849626, 0.074702550998768227, 0.073805211411634764,
         0.072931403416608628, 0.072080127805153577, 0.071250444232266827,
         0.070441466830221316, 0.069652360213760117, 0.068882335836058628,
         0.068130648659690554, 0.067396594111116953, 0.066679505290812818,
         0.065978750414207185, 0.065293730461298868, 0.064623877015205694,
         0.063968650272027958, 0.063327537206275833, 0.062700049877748992,
         0.062085723867207059, 0.061484116829451943, 0.06089480715359024,
         0.060317392721253721, 0.059751489754459056, 0.059196731745587997,
         0.058652768462681568, 0.058119265023883551, 0.057595901035437588,
         0.057082369788157505, 0.056578377507750292, 0.056083642654785618,
         0.055597895270477579, 0.055120876364782037, 0.054652337343614694,
         0.05419203947226759, 0.053739753372352525, 0.053295258549820014,
         0.052858342951806907, 0.052428802550251424, 0.052006440950377902,
         0.051591069022308984],
        [0.15384615384615385, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0.076923076923076927, 0.076923076923076927,
         0.076923076923076927, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0,0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0,0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0,0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
       ]
  },
  "neuronWeights": {
     "objectType": "dataFrameVector",
        "name": "Neuron Weights",
        "data": {
           "Neuron Weights: Input Layer – Hidden Layer 1": {
              "objectType": "dataFrame",
              "name": "Neuron Weights: Input Layer - Hidden Layer 1",
              "order": "col",
              "rowNames": ["Neuron 1", "Neuron 2", "Neuron 3", "Neuron 4"]
              "colNames": ["X1", "X2", "X3", "X4", "Weights", "Bias"],
              "colTypes": ["double", "double", "double", "double", "double", "double"],
              "indexCols": null,
              "data": [
                 [-0.75069145845261487, -0.27887307054565263, -0.79378896450938241,
                   0.093324509265532637],
                 [-0.43066792161670547, -0.5410528516228591, -0.1673473671984495,
                   0.36763405948607503],
                 [-0.66269427879746901, 0.44816597640320865, 0.022985461449100159, -
                   1.1276745743539944],
                 [ 0.01091317921324568, 0.5809695393525508, 0.023724515541765721, -
                  0.23574636017890654],
                 [-0.11534712105085403, 0.51441944511523519, -0.19651109483154025, -
                    0.49304953437134286],
                 [-0.0056227144792817983, 0.00024935739304918469, -0.0009730192415791769, -
                   0.00075515790996579716]
             ]   
           },
           "Neuron Weights": "Hidden Layer 1 - Output Layer": {
              "objectType": "dataFrame",
              "name": "Neuron Weights: Hidden Layer 1 - Output Layer",
              "order": "col",
              "rowNames": ["0", "1"],
              "colNames": ["Neuron 1", "Neuron 2", "Neuron 3", "Neuron 4", "Bias"],
              "colTypes": ["double", "double", "double", "double", "double"],
              "indexCols": null,
              "data" :[
                 [-0.77478453633089717, -0.15371632782460409],
                 [ 1.1850172724323604, -1.8065785521236266],
                 [-0.65417688123264595, -0.64439128694926806],
                 [-1.7117601902255974, 2.5736303856604104],
                 [ 0.48784434452201758, -0.48784434452201725]
              ]
           }
        }
    },

Results for numEpochsUsed, trainingTime, stoppingReason and partitionCausedStopping as requested in nncModel – actions – evaluations in the Rason model above.

    "numEpochsUsed": 100,
    "trainingTime": 0.1417815,
    "stoppingReason": "Maximum number of epochs reached",
    "partitionCausedStopping": null" 

Results for prediction contains 13 rows and 1 column. Row names are found under "rowNames", column headings under "colNames" and data is found under "data".

"trainScore": {
  "prediction": {
     "objectType": "dataFrame",
     "name": "Prediction",
     "order": "col",
     "rowNames": ["Record 1", "Record 2", "Record 3", "Record 4", "Record 5",
     "Record 6", "Record 7", "Record 8", "Record 9", "Record 10", "Record 11",
     "Record 12", "Record 13"],
     "colNames": ["Prediction: Y"],
     "colTypes": ["wstring"],
     "indexCols": null,
     "data":[
        ["0", "0", "1", "0", "1", "1", "1", "0", "0", "1", "0", "1", "1"]
     ]
  },

Results for posteriorProbability contains 13 rows and 2 columns. Row names are found under "rowNames", column headings under "colNames" and data is found under "data".

  "posteriorProbability": {
     "objectType": "dataFrame",
     "name": "posteriorProbability",
     "order": col,
     "rowNames": ["Record 1", "Record 2", "Record 3", "Record 4", "Record 5",
     "Record 6", "Record 7", "Record 8", "Record 9", "Record 10", "Record 11",
     "Record 12", "Record 13"],
     "colNames": ["0", "1"],
     "colTypes": ["double", "double"],
     "indexCols": null,
     "data": [
        [0.98142563041514053, 0.98142506620239767, 0.035388177536841307,
         0.98141295352694591, 0.036583192233183064, 0.035889568070065693,
         0.035968315202947687, 0.98142532181709696, 0.72414124228947319,
         0.035283978729137452, 0.98142313253302671, 0.035240926303000457,
         0.035240035931403735],
        [0.018574369584859543, 0.018574933797602357, 0.9646118224631588,
         0.018587046473054213, 0.96341680776681693, 0.96411043192993429,
         0.96403168479705237, 0.018574678182903086, 0.27585875771052676,
         0.96471602127086253, 0.018576867466973224, 0.96475907369699965,
         0.96475996406859632]
     ]
  },

Results for confusionMatrix contain 2 rows and 2 columns. Row names are found under "rowNames", column headings under "colNames" and data is found under "data".

  "confusionMatrix": {
    "objectType": "dataFrame",
    "name": "Confusion Matrix",
    "order": "col",
    "rowNames": ["0", "1"],
    "colNames": ["0", "1"],
    "colTypes": ["double", "double"],
    "indexCols": null,
    "data": [
      [6, 0],
      [0, 7]
    ]
  },

Results follow for accuracy, specificity, sensitivity, precision and f1.

  "accuracy": 100,
  "specificity": 1,
  "sensitivity": 1,
  "recall": 1,
  "precision": 1,
  "f1": 1
 },

Results for prediction contains 13 rows and 1 column. Row names are found under "rowNames", column headings under "colNames" and data is found under "data".

  "validScore": {
    "prediction": {
      "objectType": "dataFrame",
      "name": "prediction",
      "order": "col",
      "rowNames": ["Record 1", "Record 2", "Record 3", "Record 4", "Record 5",
      "Record 6", "Record 7", "Record 8", "Record 9", "Record 10", "Record 11",
      "Record 12", "Record 13"],
      "colNames": ["Prediction: Y"],
      "colTypes": ["wstring"],
      "indexCols": null,
      "data": [["0", "0", "1", "0", "1", "1", "1", "0", "0", "1", "0", "1", "1"]]
    },

Results for posteriorProbability contains 13 rows and 2 columns. Row names are found under "rowNames", column headings under "colNames" and data is found under "data".

  "posteriorProbability": {
     "objectType": "dataFrame",
     "name": "Posterior Probability",
     "order": col,
     "rowNames": ["Record 1", "Record 2", "Record 3", "Record 4", "Record 5", "Record 6", "Record 7", "Record 8", "Record 9", 
     "Record 10", "Record 11", "Record 12", "Record 13"],
     "colNames": ["0", "1"],
     "colTypes": ["prediction: y"],
     "indexCols": null,
     "data": [
        [0.98142563041514053, 0.98142506620239767, 0.035388177536841307,
         0.98141295352694591, 0.036583192233183064, 0.035889568070065693,
         0.035968315202947687, 0.98142532181709696, 0.72414124228947319,
         0.035283978729137452, 0.98142313253302671, 0.035240926303000457,
         0.035240035931403735],
        [0.018574369584859543, 0.018574933797602357, 0.9646118224631588,
         0.018587046473054213, 0.96341680776681693, 0.96411043192993429,
         0.96403168479705237, 0.018574678182903086, 0.27585875771052676,
         0.96471602127086253, 0.018576867466973224, 0.96475907369699965,
         0.96475996406859632]
        ]
   },

Results for confusionMatrix contain 2 rows and 2 columns. Row names are found under "rowNames", column headings under "colNames" and data is found under "data".

  "confusionMatrix": {
     "objectType": "dataFrame",
     "name": "Confusion Matrix",
     "order": "col",
     "rowNames": ["0", "1"],
     "colNames" ["0", "1"],
     "colTypes": ["double", "double"],
     "indexCols": null,
     "data": [6, 0],[0, 7]]
  },

Results for accuracy, specificity, sensitivity, precision and f1.

  "accuracy": 100,
  "specificity": 1,
  "sensitivity": 1,
  "recall": 1,
  "precision": 1,
  "f1": 1
 }
}