Evaluate data frame analytics

POST /_ml/data_frame/_evaluate

Auth

The API packages together commonly used evaluation metrics for various types of machine learning features. This has been designed for use on indexes created by data frame analytics. Evaluation requires both a ground truth field and an analytics result field to be present.

Required privileges

Cluster privileges monitor_ml

Request body

evaluation object required: Defines the type of evaluation you want to perform.
index string required: Defines the index in which the evaluation will be performed.
query {} Query: A query clause that retrieves a subset of data from the source index.

Response

application/json

classification object: Evaluation results for a classification analysis. It outputs a prediction that identifies to which of the classes each document belongs.
outlier_detection object: Evaluation results for an outlier detection analysis. It outputs the probability that each document is an outlier.
regression object: Evaluation results for a regression analysis which outputs a prediction of values.

Examples

Request Console 1 / 8 languages

Run POST _ml/data_frame/_evaluate to evaluate a a classification job for an annotated index. The actual_field contains the ground truth for classification. The predicted_field contains the predicted value calculated by the classification analysis.

Console Kibana Dev Tools

POST _ml/data_frame/_evaluate
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "predicted_field": "ml.animal_class_prediction",
      "metrics": {
        "multiclass_confusion_matrix": {}
      }
    }
  }
}

JavaScript Elasticsearch JavaScript Client

const response = await client.ml.evaluateDataFrame({
  index: "animal_classification",
  evaluation: {
    classification: {
      actual_field: "animal_class",
      predicted_field: "ml.animal_class_prediction",
      metrics: {
        multiclass_confusion_matrix: {},
      },
    },
  },
});

Python Elasticsearch Python Client

resp = client.ml.evaluate_data_frame(
    index="animal_classification",
    evaluation={
        "classification": {
            "actual_field": "animal_class",
            "predicted_field": "ml.animal_class_prediction",
            "metrics": {
                "multiclass_confusion_matrix": {}
            }
        }
    },
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ml/data_frame/_evaluate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "animal_classification",
    "evaluation": {
      "classification": {
        "actual_field": "animal_class",
        "predicted_field": "ml.animal_class_prediction",
        "metrics": {
          "multiclass_confusion_matrix": {}
        }
      }
    }
  }'

Java Elasticsearch Java Client

client.ml().evaluateDataFrame(e -> e
    .evaluation(ev -> ev
        .classification(c -> c
            .actualField("animal_class")
            .predictedField("ml.animal_class_prediction")
            .metrics(m -> m
                .multiclassConfusionMatrix(Map.of())
            )
        )
    )
    .index("animal_classification")
);

C# Elasticsearch .NET Client

var response = await client.MachineLearning
    .EvaluateDataFrameAsync(d1 => d1
        .Evaluation(d2 => d2
            .Classification(d3 => d3
                .ActualField(x => x.AnimalClass)
                .Metrics(d4 => d4
                    .MulticlassConfusionMatrix(new Dictionary<string, object>() { })
                )
                .PredictedField(x => x.Ml.AnimalClassPrediction)
            )
        )
        .Index("animal_classification")
    );

PHP Elasticsearch PHP Client

$resp = $client->ml()->evaluateDataFrame([
    "body" => [
        "index" => "animal_classification",
        "evaluation" => [
            "classification" => [
                "actual_field" => "animal_class",
                "predicted_field" => "ml.animal_class_prediction",
                "metrics" => [
                    "multiclass_confusion_matrix" => new ArrayObject([]),
                ],
            ],
        ],
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.ml.evaluate_data_frame(
  body: {
    "index": "animal_classification",
    "evaluation": {
      "classification": {
        "actual_field": "animal_class",
        "predicted_field": "ml.animal_class_prediction",
        "metrics": {
          "multiclass_confusion_matrix": {}
        }
      }
    }
  }
)

Response

A succesful response from POST _ml/data_frame/_evaluate to evaluate a classification analysis job for an annotated index. The actual_class contains the name of the class the analysis tried to predict. The actual_class_doc_count is the number of documents in the index belonging to the actual_class. The predicted_classes object contains the list of the predicted classes and the number of predictions associated with the class.

{
  "classification": {
    "multiclass_confusion_matrix": {
      "confusion_matrix": [
        {
          "actual_class": "cat",
          "actual_class_doc_count": 12,
          "predicted_classes": [
            {
              "predicted_class": "cat",
              "count": 12
            },
            {
              "predicted_class": "dog",
              "count": 0
            }
          ],
          "other_predicted_class_doc_count": 0
        },
        {
          "actual_class": "dog",
          "actual_class_doc_count": 11,
          "predicted_classes": [
            {
              "predicted_class": "dog",
              "count": 7
            },
            {
              "predicted_class": "cat",
              "count": 4
            }
          ],
          "other_predicted_class_doc_count": 0
        }
      ],
      "other_actual_class_count": 0
    }
  }
}

Request Console 1 / 8 languages

Run POST _ml/data_frame/_evaluate to evaluate a classification job with AUC ROC metrics for an annotated index. The actual_field contains the ground truth value for the actual animal classification. This is required in order to evaluate results. The class_name specifies the class name that is treated as positive during the evaluation, all the other classes are treated as negative.

Console Kibana Dev Tools

POST _ml/data_frame/_evaluate
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "metrics": {
        "auc_roc": {
          "class_name": "dog"
        }
      }
    }
  }
}

JavaScript Elasticsearch JavaScript Client

const response = await client.ml.evaluateDataFrame({
  index: "animal_classification",
  evaluation: {
    classification: {
      actual_field: "animal_class",
      metrics: {
        auc_roc: {
          class_name: "dog",
        },
      },
    },
  },
});

Python Elasticsearch Python Client

resp = client.ml.evaluate_data_frame(
    index="animal_classification",
    evaluation={
        "classification": {
            "actual_field": "animal_class",
            "metrics": {
                "auc_roc": {
                    "class_name": "dog"
                }
            }
        }
    },
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ml/data_frame/_evaluate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "animal_classification",
    "evaluation": {
      "classification": {
        "actual_field": "animal_class",
        "metrics": {
          "auc_roc": {
            "class_name": "dog"
          }
        }
      }
    }
  }'

Java Elasticsearch Java Client

client.ml().evaluateDataFrame(e -> e
    .evaluation(ev -> ev
        .classification(c -> c
            .actualField("animal_class")
            .metrics(m -> m
                .aucRoc(a -> a
                    .className("dog")
                )
            )
        )
    )
    .index("animal_classification")
);

C# Elasticsearch .NET Client

var response = await client.MachineLearning
    .EvaluateDataFrameAsync(d1 => d1
        .Evaluation(d2 => d2
            .Classification(d3 => d3
                .ActualField(x => x.AnimalClass)
                .Metrics(d4 => d4
                    .AucRoc(d5 => d5
                        .ClassName("dog")
                    )
                )
            )
        )
        .Index("animal_classification")
    );

PHP Elasticsearch PHP Client

$resp = $client->ml()->evaluateDataFrame([
    "body" => [
        "index" => "animal_classification",
        "evaluation" => [
            "classification" => [
                "actual_field" => "animal_class",
                "metrics" => [
                    "auc_roc" => [
                        "class_name" => "dog",
                    ],
                ],
            ],
        ],
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.ml.evaluate_data_frame(
  body: {
    "index": "animal_classification",
    "evaluation": {
      "classification": {
        "actual_field": "animal_class",
        "metrics": {
          "auc_roc": {
            "class_name": "dog"
          }
        }
      }
    }
  }
)

Response

A succesful response from POST _ml/data_frame/_evaluate to evaluate a classification analysis job with the AUC ROC metrics for an annotated index.

{
  "classification": {
    "auc_roc": {
      "value": 0.8941788639536681
    }
  }
}

Request Console 1 / 8 languages

Run POST _ml/data_frame/_evaluate to evaluate an outlier detection job for an annotated index.

Console Kibana Dev Tools

POST _ml/data_frame/_evaluate
{
  "index": "my_analytics_dest_index",
  "evaluation": {
    "outlier_detection": {
      "actual_field": "is_outlier",
      "predicted_probability_field": "ml.outlier_score"
    }
  }
}

JavaScript Elasticsearch JavaScript Client

const response = await client.ml.evaluateDataFrame({
  index: "my_analytics_dest_index",
  evaluation: {
    outlier_detection: {
      actual_field: "is_outlier",
      predicted_probability_field: "ml.outlier_score",
    },
  },
});

Python Elasticsearch Python Client

resp = client.ml.evaluate_data_frame(
    index="my_analytics_dest_index",
    evaluation={
        "outlier_detection": {
            "actual_field": "is_outlier",
            "predicted_probability_field": "ml.outlier_score"
        }
    },
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ml/data_frame/_evaluate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "my_analytics_dest_index",
    "evaluation": {
      "outlier_detection": {
        "actual_field": "is_outlier",
        "predicted_probability_field": "ml.outlier_score"
      }
    }
  }'

Java Elasticsearch Java Client

client.ml().evaluateDataFrame(e -> e
    .evaluation(ev -> ev
        .outlierDetection(o -> o
            .actualField("is_outlier")
            .predictedProbabilityField("ml.outlier_score")
        )
    )
    .index("my_analytics_dest_index")
);

C# Elasticsearch .NET Client

var response = await client.MachineLearning
    .EvaluateDataFrameAsync(d1 => d1
        .Evaluation(d2 => d2
            .OutlierDetection(d3 => d3
                .ActualField(x => x.IsOutlier)
                .PredictedProbabilityField(x => x.Ml.OutlierScore)
            )
        )
        .Index("my_analytics_dest_index")
    );

PHP Elasticsearch PHP Client

$resp = $client->ml()->evaluateDataFrame([
    "body" => [
        "index" => "my_analytics_dest_index",
        "evaluation" => [
            "outlier_detection" => [
                "actual_field" => "is_outlier",
                "predicted_probability_field" => "ml.outlier_score",
            ],
        ],
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.ml.evaluate_data_frame(
  body: {
    "index": "my_analytics_dest_index",
    "evaluation": {
      "outlier_detection": {
        "actual_field": "is_outlier",
        "predicted_probability_field": "ml.outlier_score"
      }
    }
  }
)

Response

A successful response from POST _ml/data_frame/_evaluate to evaluate an outlier detection job.

{
  "outlier_detection": {
    "auc_roc": {
      "value": 0.9258475774641445
    },
    "confusion_matrix": {
      "0.25": {
        "tp": 5,
        "fp": 9,
        "tn": 204,
        "fn": 5
      },
      "0.5": {
        "tp": 1,
        "fp": 5,
        "tn": 208,
        "fn": 9
      },
      "0.75": {
        "tp": 0,
        "fp": 4,
        "tn": 209,
        "fn": 10
      }
    },
    "precision": {
      "0.25": 0.35714285714285715,
      "0.5": 0.16666666666666666,
      "0.75": 0
    },
    "recall": {
      "0.25": 0.5,
      "0.5": 0.1,
      "0.75": 0
    }
  }
}

Request Console 1 / 8 languages

Run POST _ml/data_frame/_evaluate to evaluate the testing error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the test split only. The actual_field contains the ground truth for house prices. The predicted_field contains the house price calculated by the regression analysis.

Console Kibana Dev Tools

POST _ml/data_frame/_evaluate
{
  "index": "house_price_predictions",
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "ml.is_training": false
          }
        }
      ]
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {
          "offset": 10
        },
        "huber": {
          "delta": 1.5
        }
      }
    }
  }
}

JavaScript Elasticsearch JavaScript Client

const response = await client.ml.evaluateDataFrame({
  index: "house_price_predictions",
  query: {
    bool: {
      filter: [
        {
          term: {
            "ml.is_training": false,
          },
        },
      ],
    },
  },
  evaluation: {
    regression: {
      actual_field: "price",
      predicted_field: "ml.price_prediction",
      metrics: {
        r_squared: {},
        mse: {},
        msle: {
          offset: 10,
        },
        huber: {
          delta: 1.5,
        },
      },
    },
  },
});

Python Elasticsearch Python Client

resp = client.ml.evaluate_data_frame(
    index="house_price_predictions",
    query={
        "bool": {
            "filter": [
                {
                    "term": {
                        "ml.is_training": False
                    }
                }
            ]
        }
    },
    evaluation={
        "regression": {
            "actual_field": "price",
            "predicted_field": "ml.price_prediction",
            "metrics": {
                "r_squared": {},
                "mse": {},
                "msle": {
                    "offset": 10
                },
                "huber": {
                    "delta": 1.5
                }
            }
        }
    },
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ml/data_frame/_evaluate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "house_price_predictions",
    "query": {
      "bool": {
        "filter": [
          {
            "term": {
              "ml.is_training": false
            }
          }
        ]
      }
    },
    "evaluation": {
      "regression": {
        "actual_field": "price",
        "predicted_field": "ml.price_prediction",
        "metrics": {
          "r_squared": {},
          "mse": {},
          "msle": {
            "offset": 10
          },
          "huber": {
            "delta": 1.5
          }
        }
      }
    }
  }'

Java Elasticsearch Java Client

client.ml().evaluateDataFrame(e -> e
    .evaluation(ev -> ev
        .regression(r -> r
            .actualField("price")
            .predictedField("ml.price_prediction")
            .metrics(m -> m
                .mse(Map.of())
                .msle(ms -> ms
                    .offset(10.0D)
                )
                .huber(h -> h
                    .delta(1.5D)
                )
                .rSquared(Map.of())
            )
        )
    )
    .index("house_price_predictions")
    .query(q -> q
        .bool(b -> b
            .filter(f -> f
                .term(t -> t
                    .field("ml.is_training")
                    .value(FieldValue.of(false))
                )
            )
        )
    )
);

C# Elasticsearch .NET Client

var response = await client.MachineLearning
    .EvaluateDataFrameAsync(d1 => d1
        .Evaluation(d2 => d2
            .Regression(d3 => d3
                .ActualField(x => x.Price)
                .Metrics(d4 => d4
                    .Huber(d5 => d5
                        .Delta(1.5d)
                    )
                    .Mse(new Dictionary<string, object>() { })
                    .Msle(d5 => d5
                        .Offset(10d)
                    )
                    .RSquared(new Dictionary<string, object>() { })
                )
                .PredictedField(x => x.Ml.PricePrediction)
            )
        )
        .Index("house_price_predictions")
        .Query(d2 => d2
            .Bool(d3 => d3
                .Filter(d4 => d4
                    .Term(d5 => d5
                        .Field(x => x.Ml.IsTraining)
                        .Value(false)
                    )
                )
            )
        )
    );

PHP Elasticsearch PHP Client

$resp = $client->ml()->evaluateDataFrame([
    "body" => [
        "index" => "house_price_predictions",
        "query" => [
            "bool" => [
                "filter" => array(
                    [
                        "term" => [
                            "ml.is_training" => false,
                        ],
                    ],
                ),
            ],
        ],
        "evaluation" => [
            "regression" => [
                "actual_field" => "price",
                "predicted_field" => "ml.price_prediction",
                "metrics" => [
                    "r_squared" => new ArrayObject([]),
                    "mse" => new ArrayObject([]),
                    "msle" => [
                        "offset" => 10,
                    ],
                    "huber" => [
                        "delta" => 1.5,
                    ],
                ],
            ],
        ],
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.ml.evaluate_data_frame(
  body: {
    "index": "house_price_predictions",
    "query": {
      "bool": {
        "filter": [
          {
            "term": {
              "ml.is_training": false
            }
          }
        ]
      }
    },
    "evaluation": {
      "regression": {
        "actual_field": "price",
        "predicted_field": "ml.price_prediction",
        "metrics": {
          "r_squared": {},
          "mse": {},
          "msle": {
            "offset": 10
          },
          "huber": {
            "delta": 1.5
          }
        }
      }
    }
  }
)

Response

No example

Request Console 1 / 8 languages

Run POST _ml/data_frame/_evaluate to evaluate the training error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the training split only. The actual_field contains the ground truth for house prices. The predicted_field contains the house price calculated by the regression analysis.

Console Kibana Dev Tools

POST _ml/data_frame/_evaluate
{
  "index": "house_price_predictions",
  "query": {
    "term": {
      "ml.is_training": {
        "value": true
      }
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {},
        "huber": {}
      }
    }
  }
}

JavaScript Elasticsearch JavaScript Client

const response = await client.ml.evaluateDataFrame({
  index: "house_price_predictions",
  query: {
    term: {
      "ml.is_training": {
        value: true,
      },
    },
  },
  evaluation: {
    regression: {
      actual_field: "price",
      predicted_field: "ml.price_prediction",
      metrics: {
        r_squared: {},
        mse: {},
        msle: {},
        huber: {},
      },
    },
  },
});

Python Elasticsearch Python Client

resp = client.ml.evaluate_data_frame(
    index="house_price_predictions",
    query={
        "term": {
            "ml.is_training": {
                "value": True
            }
        }
    },
    evaluation={
        "regression": {
            "actual_field": "price",
            "predicted_field": "ml.price_prediction",
            "metrics": {
                "r_squared": {},
                "mse": {},
                "msle": {},
                "huber": {}
            }
        }
    },
)

curl Shell

curl -X POST "$ELASTICSEARCH_URL/_ml/data_frame/_evaluate" \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "house_price_predictions",
    "query": {
      "term": {
        "ml.is_training": {
          "value": true
        }
      }
    },
    "evaluation": {
      "regression": {
        "actual_field": "price",
        "predicted_field": "ml.price_prediction",
        "metrics": {
          "r_squared": {},
          "mse": {},
          "msle": {},
          "huber": {}
        }
      }
    }
  }'

Java Elasticsearch Java Client

client.ml().evaluateDataFrame(e -> e
    .evaluation(ev -> ev
        .regression(r -> r
            .actualField("price")
            .predictedField("ml.price_prediction")
            .metrics(m -> m
                .mse(Map.of())
                .msle(ms -> ms)
                .huber(h -> h)
                .rSquared(Map.of())
            )
        )
    )
    .index("house_price_predictions")
    .query(q -> q
        .term(t -> t
            .field("ml.is_training")
            .value(FieldValue.of(true))
        )
    )
);

C# Elasticsearch .NET Client

var response = await client.MachineLearning
    .EvaluateDataFrameAsync(d1 => d1
        .Evaluation(d2 => d2
            .Regression(d3 => d3
                .ActualField(x => x.Price)
                .Metrics(d4 => d4
                    .Huber()
                    .Mse(new Dictionary<string, object>() { })
                    .Msle()
                    .RSquared(new Dictionary<string, object>() { })
                )
                .PredictedField(x => x.Ml.PricePrediction)
            )
        )
        .Index("house_price_predictions")
        .Query(d2 => d2
            .Term(d3 => d3
                .Field(x => x.Ml.IsTraining)
                .Value(true)
            )
        )
    );

PHP Elasticsearch PHP Client

$resp = $client->ml()->evaluateDataFrame([
    "body" => [
        "index" => "house_price_predictions",
        "query" => [
            "term" => [
                "ml.is_training" => [
                    "value" => true,
                ],
            ],
        ],
        "evaluation" => [
            "regression" => [
                "actual_field" => "price",
                "predicted_field" => "ml.price_prediction",
                "metrics" => [
                    "r_squared" => new ArrayObject([]),
                    "mse" => new ArrayObject([]),
                    "msle" => new ArrayObject([]),
                    "huber" => new ArrayObject([]),
                ],
            ],
        ],
    ],
]);

Ruby Elasticsearch Ruby Client

response = client.ml.evaluate_data_frame(
  body: {
    "index": "house_price_predictions",
    "query": {
      "term": {
        "ml.is_training": {
          "value": true
        }
      }
    },
    "evaluation": {
      "regression": {
        "actual_field": "price",
        "predicted_field": "ml.price_prediction",
        "metrics": {
          "r_squared": {},
          "mse": {},
          "msle": {},
          "huber": {}
        }
      }
    }
  }
)

Response

No example