TFMA Metrics

tensorflow_model_analysis.metrics

Init module for TensorFlow Model Analysis metrics.

Attributes

MetricComputations module-attribute

MetricComputations = List[
    Union[
        MetricComputation,
        DerivedMetricComputation,
        CrossSliceMetricComputation,
        CIDerivedMetricComputation,
    ]
]

MetricsDict module-attribute

MetricsDict = Dict[MetricKey, MetricValueType]

NO_PREDICTED_CLASS_ID module-attribute

NO_PREDICTED_CLASS_ID = -1

Classes

AUC

AUC(
    num_thresholds: Optional[int] = None,
    curve: str = "ROC",
    summation_method: str = "interpolation",
    name: Optional[str] = None,
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetricBase

Approximates the AUC (Area under the curve) of the ROC or PR curves.

The AUC (Area under the curve) of the ROC (Receiver operating characteristic; default) or PR (Precision Recall) curves are quality measures of binary classifiers. Unlike the accuracy, and like cross-entropy losses, ROC-AUC and PR-AUC evaluate all the operational points of a model.

This class approximates AUCs using a Riemann sum. During the metric accumulation phase, predictions are accumulated within predefined buckets by value. The AUC is then computed by interpolating per-bucket averages. These buckets define the evaluated operational points.

This metric uses true_positives, true_negatives, false_positives and false_negatives to compute the AUC. To discretize the AUC curve, a linearly spaced set of thresholds is used to compute pairs of recall and precision values. The area under the ROC-curve is therefore computed using the height of the recall values by the false positive rate, while the area under the PR-curve is the computed using the height of the precision values by the recall.

This value is ultimately returned as auc, an idempotent operation that computes the area under a discretized curve of precision versus recall values (computed using the aforementioned variables). The num_thresholds variable controls the degree of discretization with larger numbers of thresholds more closely approximating the true AUC. The quality of the approximation may vary dramatically depending on num_thresholds. The thresholds parameter can be used to manually specify thresholds which split the predictions more evenly.

For a best approximation of the real AUC, predictions should be distributed approximately uniformly in the range [0, 1]. The quality of the AUC approximation may be poor if this is not the case. Setting summation_method to 'minoring' or 'majoring' can help quantify the error in the approximation by providing lower or upper bound estimate of the AUC.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes AUC metric.

PARAMETER	DESCRIPTION
num_thresholds	(Optional) Defaults to 10000. The number of thresholds to use when discretizing the roc curve. Values must be > 1. TYPE: Optional[int] DEFAULT: None
curve	(Optional) Specifies the name of the curve to be computed, 'ROC' [default] or 'PR' for the Precision-Recall-curve. TYPE: str DEFAULT: 'ROC'
summation_method	(Optional) Specifies the Riemann summation method used. 'interpolation' (default) applies mid-point summation scheme for ROC. For PR-AUC, interpolates (true/false) positives but not the ratio that is precision (see Davis & Goadrich 2006 for details); 'minoring' applies left summation for increasing intervals and right summation for decreasing intervals; 'majoring' does the opposite. TYPE: str DEFAULT: 'interpolation'
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
thresholds	(Optional) A list of floating point values to use as the thresholds for discretizing the curve. If set, the num_thresholds parameter is ignored. Values should be in [0, 1]. Endpoint thresholds equal to {-epsilon, 1+epsilon} for a small positive epsilon value will be automatically included with these to correctly handle predictions equal to exactly 0 or 1. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             num_thresholds: Optional[int] = None,
             curve: str = 'ROC',
             summation_method: str = 'interpolation',
             name: Optional[str] = None,
             thresholds: Optional[Union[float, List[float]]] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes AUC metric.

  Args:
    num_thresholds: (Optional) Defaults to 10000. The number of thresholds to
      use when discretizing the roc curve. Values must be > 1.
    curve: (Optional) Specifies the name of the curve to be computed, 'ROC'
      [default] or 'PR' for the Precision-Recall-curve.
    summation_method: (Optional) Specifies the [Riemann summation method](
      https://en.wikipedia.org/wiki/Riemann_sum) used. 'interpolation'
        (default) applies mid-point summation scheme for `ROC`. For PR-AUC,
        interpolates (true/false) positives but not the ratio that is
        precision (see Davis & Goadrich 2006 for details); 'minoring' applies
        left summation for increasing intervals and right summation for
        decreasing intervals; 'majoring' does the opposite.
    name: (Optional) string name of the metric instance.
    thresholds: (Optional) A list of floating point values to use as the
      thresholds for discretizing the curve. If set, the `num_thresholds`
      parameter is ignored. Values should be in [0, 1]. Endpoint thresholds
      equal to {-epsilon, 1+epsilon} for a small positive epsilon value will
      be automatically included with these to correctly handle predictions
      equal to exactly 0 or 1.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      num_thresholds=num_thresholds,
      thresholds=thresholds,
      curve=curve,
      summation_method=summation_method,
      name=name,
      top_k=top_k,
      class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

AUCCurve

Bases: Enum

Attributes

PR class-attribute instance-attribute

PR = 'PR'

ROC class-attribute instance-attribute

ROC = 'ROC'

AUCPrecisionRecall

AUCPrecisionRecall(
    num_thresholds: Optional[int] = None,
    summation_method: str = "interpolation",
    name: Optional[str] = None,
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: AUC

Alias for AUC(curve='PR').

Initializes AUCPrecisionRecall metric.

PARAMETER	DESCRIPTION
num_thresholds	(Optional) Defaults to 10000. The number of thresholds to use when discretizing the roc curve. Values must be > 1. TYPE: Optional[int] DEFAULT: None
summation_method	(Optional) Specifies the Riemann summation method used. 'interpolation' interpolates (true/false) positives but not the ratio that is precision (see Davis & Goadrich 2006 for details); 'minoring' applies left summation for increasing intervals and right summation for decreasing intervals; 'majoring' does the opposite. TYPE: str DEFAULT: 'interpolation'
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
thresholds	(Optional) A list of floating point values to use as the thresholds for discretizing the curve. If set, the num_thresholds parameter is ignored. Values should be in [0, 1]. Endpoint thresholds equal to {-epsilon, 1+epsilon} for a small positive epsilon value will be automatically included with these to correctly handle predictions equal to exactly 0 or 1. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             num_thresholds: Optional[int] = None,
             summation_method: str = 'interpolation',
             name: Optional[str] = None,
             thresholds: Optional[Union[float, List[float]]] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes AUCPrecisionRecall metric.

  Args:
    num_thresholds: (Optional) Defaults to 10000. The number of thresholds to
      use when discretizing the roc curve. Values must be > 1.
    summation_method: (Optional) Specifies the [Riemann summation method](
      https://en.wikipedia.org/wiki/Riemann_sum) used. 'interpolation'
        interpolates (true/false) positives but not the ratio that is
        precision (see Davis & Goadrich 2006 for details); 'minoring' applies
        left summation for increasing intervals and right summation for
        decreasing intervals; 'majoring' does the opposite.
    name: (Optional) string name of the metric instance.
    thresholds: (Optional) A list of floating point values to use as the
      thresholds for discretizing the curve. If set, the `num_thresholds`
      parameter is ignored. Values should be in [0, 1]. Endpoint thresholds
      equal to {-epsilon, 1+epsilon} for a small positive epsilon value will
      be automatically included with these to correctly handle predictions
      equal to exactly 0 or 1.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      num_thresholds=num_thresholds,
      thresholds=thresholds,
      curve='PR',
      summation_method=summation_method,
      name=name,
      top_k=top_k,
      class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Remove the irrelevant 'curve' keyword inherited from parent class AUC().
  # This is needed when the __init__ of the child class has a different set of
  # kwargs than that of its parent class.
  result = super().get_config()
  del result['curve']
  return result

AUCSummationMethod

Bases: Enum

Attributes

INTERPOLATION class-attribute instance-attribute

INTERPOLATION = 'interpolation'

MAJORING class-attribute instance-attribute

MAJORING = 'majoring'

MINORING class-attribute instance-attribute

MINORING = 'minoring'

AttributionsMetric

AttributionsMetric(
    create_computations_fn: Callable[
        ..., MetricComputations
    ],
    **kwargs,
)

Bases: Metric

Base type for attribution metrics.

Initializes metric.

PARAMETER	DESCRIPTION
create_computations_fn	Function to create the metrics computations (e.g. mean_label, etc). This function should take the args passed to init as as input along with any of eval_config, schema, model_names, output_names, sub_keys, aggregation_type, or query_key (where needed). TYPE: Callable[..., MetricComputations]
**kwargs	Any additional kwargs to pass to create_computations_fn. These should only contain primitive types or lists/dicts of primitive types. The kwargs passed to computations have precendence over these kwargs. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/metric_types.py

def __init__(
    self, create_computations_fn: Callable[..., MetricComputations], **kwargs
):
  """Initializes metric.

  Args:
    create_computations_fn: Function to create the metrics computations (e.g.
      mean_label, etc). This function should take the args passed to __init__
      as as input along with any of eval_config, schema, model_names,
      output_names, sub_keys, aggregation_type, or query_key (where needed).
    **kwargs: Any additional kwargs to pass to create_computations_fn. These
      should only contain primitive types or lists/dicts of primitive types.
      The kwargs passed to computations have precendence over these kwargs.
  """
  self.create_computations_fn = create_computations_fn
  if 'name' in kwargs:
    if not kwargs['name'] and self._default_name():
      kwargs['name'] = self._default_name()  # pylint: disable=assignment-from-none
    name = kwargs['name']
  else:
    name = None
  self.name = name
  self.kwargs = kwargs
  if hasattr(inspect, 'getfullargspec'):
    self._args = inspect.getfullargspec(self.create_computations_fn).args
  else:
    self._args = inspect.getargspec(self.create_computations_fn).args  # pylint: disable=deprecated-method

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

BalancedAccuracy

BalancedAccuracy(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Balanced accuracy (BA).

Initializes balanced accuracy.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes balanced accuracy.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  tpr_denominator = tp + fn
  tnr_denominator = tn + fp
  if tpr_denominator > 0.0 and tnr_denominator > 0.0:
    tpr = tp / tpr_denominator
    tnr = tn / tnr_denominator
    return (tpr + tnr) / 2
  else:
    return float('nan')

BinaryAccuracy

BinaryAccuracy(
    threshold: Optional[float] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
)

Bases: ConfusionMatrixMetric

Calculates how often predictions match binary labels.

This metric computes the accuracy based on (TP + TN) / (TP + FP + TN + FN).

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes BinaryAccuracy metric.

PARAMETER	DESCRIPTION
threshold	(Optional) A float value in [0, 1]. The threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). If neither threshold nor top_k are set, the default is to calculate with threshold=0.5. TYPE: Optional[float] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             threshold: Optional[float] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None):
  """Initializes BinaryAccuracy metric.

  Args:
    threshold: (Optional) A float value in [0, 1]. The threshold is compared
      with prediction values to determine the truth value of predictions
      (i.e., above the threshold is `true`, below is `false`). If neither
      threshold nor top_k are set, the default is to calculate with
      `threshold=0.5`.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
  """
  super().__init__(
      thresholds=threshold, top_k=top_k, class_id=class_id, name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return _divide_only_positive_denominator(tp + tn, tp + fp + tn + fn)

BinaryCrossEntropy

BinaryCrossEntropy(
    name: str = BINARY_CROSSENTROPY_NAME,
    from_logits: bool = False,
    label_smoothing: float = 0.0,
)

Bases: Metric

Calculates the binary cross entropy.

The metric computes the cross entropy when there are only two label classes (0 and 1). See definition at: https://en.wikipedia.org/wiki/Cross_entropy

Initializes binary cross entropy metric.

PARAMETER	DESCRIPTION
name	The name of the metric. TYPE: str DEFAULT: BINARY_CROSSENTROPY_NAME
from_logits	(Optional) Whether output is expected to be a logits tensor. By default, we consider that output encodes a probability distribution. TYPE: bool DEFAULT: False
label_smoothing	Float in [0, 1]. If > 0 then smooth the labels by squeezing them towards 0.5 That is, using 1. - 0.5 * label_smoothing for the target class and 0.5 * label_smoothing for the non-target class. TYPE: float DEFAULT: 0.0

Source code in tensorflow_model_analysis/metrics/cross_entropy_metrics.py

def __init__(
    self,
    name: str = BINARY_CROSSENTROPY_NAME,
    from_logits: bool = False,
    label_smoothing: float = 0.0,
):
  """Initializes binary cross entropy metric.

  Args:
    name: The name of the metric.
    from_logits: (Optional) Whether output is expected to be a logits tensor.
      By default, we consider that output encodes a probability distribution.
    label_smoothing: Float in [0, 1]. If > `0` then smooth the labels by
      squeezing them towards 0.5 That is, using `1. - 0.5 * label_smoothing`
      for the target class and `0.5 * label_smoothing` for the non-target
      class.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _binary_cross_entropy_computations
      ),
      name=name,
      from_logits=from_logits,
      label_smoothing=label_smoothing,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

BooleanFlipRates

BooleanFlipRates(
    threshold: float = _DEFAULT_FLIP_RATE_THRESHOLD,
    flip_rate_name: str = FLIP_RATE_NAME,
    neg_to_neg_flip_rate_name: str = NEG_TO_NEG_FLIP_RATE_NAME,
    neg_to_pos_flip_rate_name: str = NEG_TO_POS_FLIP_RATE_NAME,
    pos_to_neg_flip_rate_name: str = POS_TO_NEG_FLIP_RATE_NAME,
    pos_to_pos_flip_rate_name: str = POS_TO_POS_FLIP_RATE_NAME,
)

Bases: Metric

FlipRate is the rate at which predictions between models switch.

Given a pair of models and a threshold for converting continuous model outputs into boolean predictions, this metric will produce three numbers (keyed by separate MetricKeys):

• (symmetric) flip rate: The number of times the boolean predictions don't match, regardless of the direction of the flip.
• negative-to-positive flip rate: The rate at which the baseline model's boolean prediction is negative but the candidate model's is positive.
• positive-to-negative flip rate: The rate at which the baseline model's boolean prediction is positive but the candidate model's is negative.

Initializes BooleanFlipRates metric.

PARAMETER	DESCRIPTION
threshold	The threshold to use for converting the model prediction into a boolean value that can be used for comparison between models. TYPE: float DEFAULT: _DEFAULT_FLIP_RATE_THRESHOLD
flip_rate_name	Metric name for symmetric flip rate. TYPE: str DEFAULT: FLIP_RATE_NAME
neg_to_neg_flip_rate_name	Metric name for the negative-to-negative flip rate. TYPE: str DEFAULT: NEG_TO_NEG_FLIP_RATE_NAME
neg_to_pos_flip_rate_name	Metric name for the negative-to-positive flip rate. TYPE: str DEFAULT: NEG_TO_POS_FLIP_RATE_NAME
pos_to_neg_flip_rate_name	Metric name for the positive-to-negative flip rate. TYPE: str DEFAULT: POS_TO_NEG_FLIP_RATE_NAME
pos_to_pos_flip_rate_name	Metric name for the positive-to-positive flip rate. TYPE: str DEFAULT: POS_TO_POS_FLIP_RATE_NAME

Source code in tensorflow_model_analysis/metrics/flip_metrics.py

def __init__(
    self,
    threshold: float = _DEFAULT_FLIP_RATE_THRESHOLD,
    flip_rate_name: str = FLIP_RATE_NAME,
    neg_to_neg_flip_rate_name: str = NEG_TO_NEG_FLIP_RATE_NAME,
    neg_to_pos_flip_rate_name: str = NEG_TO_POS_FLIP_RATE_NAME,
    pos_to_neg_flip_rate_name: str = POS_TO_NEG_FLIP_RATE_NAME,
    pos_to_pos_flip_rate_name: str = POS_TO_POS_FLIP_RATE_NAME,
):
  """Initializes BooleanFlipRates metric.

  Args:
    threshold: The threshold to use for converting the model prediction into a
      boolean value that can be used for comparison between models.
    flip_rate_name: Metric name for symmetric flip rate.
    neg_to_neg_flip_rate_name: Metric name for the negative-to-negative flip
      rate.
    neg_to_pos_flip_rate_name: Metric name for the negative-to-positive flip
      rate.
    pos_to_neg_flip_rate_name: Metric name for the positive-to-negative flip
      rate.
    pos_to_pos_flip_rate_name: Metric name for the positive-to-positive flip
      rate.
  """

  super().__init__(
      _boolean_flip_rates_computations,
      symmetric_flip_rate_name=flip_rate_name,
      neg_to_neg_flip_rate_name=neg_to_neg_flip_rate_name,
      neg_to_pos_flip_rate_name=neg_to_pos_flip_rate_name,
      pos_to_neg_flip_rate_name=pos_to_neg_flip_rate_name,
      pos_to_pos_flip_rate_name=pos_to_pos_flip_rate_name,
      threshold=threshold,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

COCOAveragePrecision

COCOAveragePrecision(
    num_thresholds: Optional[int] = None,
    iou_threshold: Optional[float] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    recalls: Optional[List[float]] = None,
    num_recalls: Optional[int] = None,
    name: Optional[str] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: Metric

Confusion matrix at thresholds.

It computes the average precision of object detections for a single class and a single iou_threshold.

Initialize average precision metric.

This metric is only used in object-detection setting. It does not support sub_key parameters due to the matching algorithm of bounding boxes.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
num_thresholds	(Optional) Number of thresholds to use for calculating the matrices and finding the precision at given recall. TYPE: Optional[int] DEFAULT: None
iou_threshold	(Optional) Threholds for a detection and ground truth pair with specific iou to be considered as a match. TYPE: Optional[float] DEFAULT: None
class_id	(Optional) The class id for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class id. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) The area-range for objects to be considered for metrics. TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. TYPE: Optional[int] DEFAULT: None
recalls	(Optional) recalls at which precisions will be calculated. TYPE: Optional[List[float]] DEFAULT: None
num_recalls	(Optional) Used for objecth detection, the number of recalls for calculating average precision, it equally generates points bewteen 0 and 1. (Only one of recalls and num_recalls should be used). TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_metrics.py

def __init__(self,
             num_thresholds: Optional[int] = None,
             iou_threshold: Optional[float] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             recalls: Optional[List[float]] = None,
             num_recalls: Optional[int] = None,
             name: Optional[str] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initialize average precision metric.

  This metric is only used in object-detection setting. It does not support
  sub_key parameters due to the matching algorithm of bounding boxes.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    num_thresholds: (Optional) Number of thresholds to use for calculating the
      matrices and finding the precision at given recall.
    iou_threshold: (Optional) Threholds for a detection and ground truth pair
      with specific iou to be considered as a match.
    class_id: (Optional) The class id for calculating metrics.
    class_weight: (Optional) The weight associated with the object class id.
    area_range: (Optional) The area-range for objects to be considered for
      metrics.
    max_num_detections: (Optional) The maximum number of detections for a
      single image.
    recalls: (Optional) recalls at which precisions will be calculated.
    num_recalls: (Optional) Used for objecth detection, the number of recalls
      for calculating average precision, it equally generates points bewteen 0
      and 1. (Only one of recalls and num_recalls should be used).
    name: (Optional) string name of the metric instance.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """
  if recalls is not None:
    recall_thresholds = recalls
  elif num_recalls is not None:
    recall_thresholds = np.linspace(0.0, 1.0, num_recalls)
  else:
    # by default set recall_thresholds to [0.0:0.01:1.0].
    recall_thresholds = np.linspace(0.0, 1.0, 101)

  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      num_thresholds=num_thresholds,
      iou_threshold=iou_threshold,
      class_id=class_id,
      class_weight=class_weight,
      area_range=area_range,
      max_num_detections=max_num_detections,
      recall_thresholds=recall_thresholds,
      name=name,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

COCOAverageRecall

COCOAverageRecall(
    iou_thresholds: Optional[List[float]] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    name: Optional[str] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: Metric

Average recall metric for object detection.

It computes the average precision metric for object detections for a single class. It averages MaxRecall metric over mulitple IoU thresholds.

Initializes average recall metric.

This metric is only used in object-detection setting. It does not support sub_key parameters due to the matching algorithm of bounding boxes.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
iou_thresholds	(Optional) Threholds for a detection and ground truth pair with specific iou to be considered as a match. TYPE: Optional[List[float]] DEFAULT: None
class_id	(Optional) The class ids for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class ids. If it is provided, it should have the same length as class_ids. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) The area-range for objects to be considered for metrics. TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. TYPE: Optional[int] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_metrics.py

def __init__(self,
             iou_thresholds: Optional[List[float]] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             name: Optional[str] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes average recall metric.

  This metric is only used in object-detection setting. It does not support
  sub_key parameters due to the matching algorithm of bounding boxes.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    iou_thresholds: (Optional) Threholds for a detection and ground truth pair
      with specific iou to be considered as a match.
    class_id: (Optional) The class ids for calculating metrics.
    class_weight: (Optional) The weight associated with the object class ids.
      If it is provided, it should have the same length as class_ids.
    area_range: (Optional) The area-range for objects to be considered for
      metrics.
    max_num_detections: (Optional) The maximum number of detections for a
      single image.
    name: (Optional) Metric name.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """

  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      iou_thresholds=iou_thresholds,
      class_id=class_id,
      class_weight=class_weight,
      area_range=area_range,
      max_num_detections=max_num_detections,
      name=name,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

COCOMeanAveragePrecision

COCOMeanAveragePrecision(
    num_thresholds: Optional[int] = None,
    iou_thresholds: Optional[List[float]] = None,
    class_ids: Optional[List[int]] = None,
    class_weights: Optional[List[float]] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    recalls: Optional[List[float]] = None,
    num_recalls: Optional[int] = None,
    name: Optional[str] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: Metric

Mean average precision for object detections.

It calculates the mean average precision metric for object detections. It averages COCOAveragePrecision over multiple classes and IoU thresholds.

Initializes mean average precision metric.

This metric is only used in object-detection setting. It does not support sub_key parameters due to the matching algorithm of bounding boxes.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
num_thresholds	(Optional) Number of thresholds to use for calculating the matrices and finding the precision at given recall. TYPE: Optional[int] DEFAULT: None
iou_thresholds	(Optional) Threholds for a detection and ground truth pair with specific iou to be considered as a match. TYPE: Optional[List[float]] DEFAULT: None
class_ids	(Optional) The class ids for calculating metrics. TYPE: Optional[List[int]] DEFAULT: None
class_weights	(Optional) The weight associated with the object class ids. If it is provided, it should have the same length as class_ids. TYPE: Optional[List[float]] DEFAULT: None
area_range	(Optional) The area-range for objects to be considered for metrics. TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. TYPE: Optional[int] DEFAULT: None
recalls	(Optional) recalls at which precisions will be calculated. TYPE: Optional[List[float]] DEFAULT: None
num_recalls	(Optional) Used for objecth detection, the number of recalls for calculating average precision, it equally generates points bewteen 0 and 1. (Only one of recalls and num_recalls should be used). TYPE: Optional[int] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_metrics.py

def __init__(self,
             num_thresholds: Optional[int] = None,
             iou_thresholds: Optional[List[float]] = None,
             class_ids: Optional[List[int]] = None,
             class_weights: Optional[List[float]] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             recalls: Optional[List[float]] = None,
             num_recalls: Optional[int] = None,
             name: Optional[str] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes mean average precision metric.

  This metric is only used in object-detection setting. It does not support
  sub_key parameters due to the matching algorithm of bounding boxes.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    num_thresholds: (Optional) Number of thresholds to use for calculating the
      matrices and finding the precision at given recall.
    iou_thresholds: (Optional) Threholds for a detection and ground truth pair
      with specific iou to be considered as a match.
    class_ids: (Optional) The class ids for calculating metrics.
    class_weights: (Optional) The weight associated with the object class ids.
      If it is provided, it should have the same length as class_ids.
    area_range: (Optional) The area-range for objects to be considered for
      metrics.
    max_num_detections: (Optional) The maximum number of detections for a
      single image.
    recalls: (Optional) recalls at which precisions will be calculated.
    num_recalls: (Optional) Used for objecth detection, the number of recalls
      for calculating average precision, it equally generates points bewteen 0
      and 1. (Only one of recalls and num_recalls should be used).
    name: (Optional) Metric name.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """

  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      num_thresholds=num_thresholds,
      iou_thresholds=iou_thresholds,
      class_ids=class_ids,
      class_weights=class_weights,
      area_range=area_range,
      max_num_detections=max_num_detections,
      recalls=recalls,
      num_recalls=num_recalls,
      name=name,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

COCOMeanAverageRecall

COCOMeanAverageRecall(
    iou_thresholds: Optional[List[float]] = None,
    class_ids: Optional[List[int]] = None,
    class_weights: Optional[List[float]] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    name: Optional[str] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: Metric

Mean Average recall metric for object detection.

It computes the mean average precision metric for object detections for a single class. It averages COCOAverageRecall metric over mulitple classes.

Initializes average recall metric.

This metric is only used in object-detection setting. It does not support sub_key parameters due to the matching algorithm of bounding boxes.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
iou_thresholds	(Optional) Threholds for a detection and ground truth pair with specific iou to be considered as a match. TYPE: Optional[List[float]] DEFAULT: None
class_ids	(Optional) The class ids for calculating metrics. TYPE: Optional[List[int]] DEFAULT: None
class_weights	(Optional) The weight associated with the object class ids. If it is provided, it should have the same length as class_ids. TYPE: Optional[List[float]] DEFAULT: None
area_range	(Optional) The area-range for objects to be considered for metrics. TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. TYPE: Optional[int] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_metrics.py

def __init__(self,
             iou_thresholds: Optional[List[float]] = None,
             class_ids: Optional[List[int]] = None,
             class_weights: Optional[List[float]] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             name: Optional[str] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes average recall metric.

  This metric is only used in object-detection setting. It does not support
  sub_key parameters due to the matching algorithm of bounding boxes.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    iou_thresholds: (Optional) Threholds for a detection and ground truth pair
      with specific iou to be considered as a match.
    class_ids: (Optional) The class ids for calculating metrics.
    class_weights: (Optional) The weight associated with the object class ids.
      If it is provided, it should have the same length as class_ids.
    area_range: (Optional) The area-range for objects to be considered for
      metrics.
    max_num_detections: (Optional) The maximum number of detections for a
      single image.
    name: (Optional) Metric name.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """

  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      iou_thresholds=iou_thresholds,
      class_ids=class_ids,
      class_weights=class_weights,
      area_range=area_range,
      max_num_detections=max_num_detections,
      name=name,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

Calibration

Calibration(name: str = CALIBRATION_NAME)

Bases: Metric

Calibration.

Calibration in this context is defined as the total weighted predictions / total weighted labels.

Initializes calibration.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: CALIBRATION_NAME

Source code in tensorflow_model_analysis/metrics/calibration.py

def __init__(self, name: str = CALIBRATION_NAME):
  """Initializes calibration.

  Args:
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(_calibration), name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

CalibrationPlot

CalibrationPlot(
    num_buckets: int = DEFAULT_NUM_BUCKETS,
    left: Optional[float] = None,
    right: Optional[float] = None,
    name: str = CALIBRATION_PLOT_NAME,
)

Bases: Metric

Calibration plot.

Initializes calibration plot.

PARAMETER	DESCRIPTION
num_buckets	Number of buckets to use when creating the plot. Defaults to 1000. TYPE: int DEFAULT: DEFAULT_NUM_BUCKETS
left	Left boundary of plot. Defaults to 0.0 when a schema is not provided. TYPE: Optional[float] DEFAULT: None
right	Right boundary of plot. Defaults to 1.0 when a schema is not provided. TYPE: Optional[float] DEFAULT: None
name	Plot name. TYPE: str DEFAULT: CALIBRATION_PLOT_NAME

Source code in tensorflow_model_analysis/metrics/calibration_plot.py

def __init__(self,
             num_buckets: int = DEFAULT_NUM_BUCKETS,
             left: Optional[float] = None,
             right: Optional[float] = None,
             name: str = CALIBRATION_PLOT_NAME):
  """Initializes calibration plot.

  Args:
    num_buckets: Number of buckets to use when creating the plot. Defaults to
      1000.
    left: Left boundary of plot. Defaults to 0.0 when a schema is not
      provided.
    right: Right boundary of plot. Defaults to 1.0 when a schema is not
      provided.
    name: Plot name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(_calibration_plot),
      num_buckets=num_buckets,
      left=left,
      right=right,
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

CategoricalCrossEntropy

CategoricalCrossEntropy(
    name: str = CATEGORICAL_CROSSENTROPY_NAME,
    from_logits: bool = False,
    label_smoothing: float = 0.0,
)

Bases: Metric

Calculates the categorical cross entropy.

The metric computes the cross entropy when there are multiple classes. It outputs a numpy array.

Initializes categorical cross entropy metric.

PARAMETER	DESCRIPTION
name	The name of the metric. TYPE: str DEFAULT: CATEGORICAL_CROSSENTROPY_NAME
from_logits	(Optional) Whether output is expected to be a logits tensor. By default, we consider that output encodes a probability distribution. TYPE: bool DEFAULT: False
label_smoothing	Float in [0, 1]. If > 0 then smooth the labels. For example, if 0.1, use 0.1 / num_classes for non-target labels and 0.9 + 0.1 / num_classes for target labels. TYPE: float DEFAULT: 0.0

Source code in tensorflow_model_analysis/metrics/cross_entropy_metrics.py

def __init__(
    self,
    name: str = CATEGORICAL_CROSSENTROPY_NAME,
    from_logits: bool = False,
    label_smoothing: float = 0.0,
):
  """Initializes categorical cross entropy metric.

  Args:
    name: The name of the metric.
    from_logits: (Optional) Whether output is expected to be a logits tensor.
      By default, we consider that output encodes a probability distribution.
    label_smoothing: Float in [0, 1]. If > `0` then smooth the labels. For
      example, if `0.1`, use `0.1 / num_classes` for non-target labels and
      `0.9 + 0.1 / num_classes` for target labels.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _categorical_cross_entropy_computations
      ),
      name=name,
      from_logits=from_logits,
      label_smoothing=label_smoothing,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

CoefficientOfDiscrimination

CoefficientOfDiscrimination(
    name: str = COEFFICIENT_OF_DISCRIMINATION_NAME,
)

Bases: Metric

Coefficient of discrimination metric.

The coefficient of discrimination measures the differences between the average prediction for the positive examples and the average prediction for the negative examples.

The formula is: AVG(pred | label = 1) - AVG(pred | label = 0) More details can be found in the following paper: https://www.tandfonline.com/doi/abs/10.1198/tast.2009.08210

Initializes coefficient of discrimination metric.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: COEFFICIENT_OF_DISCRIMINATION_NAME

Source code in tensorflow_model_analysis/metrics/tjur_discrimination.py

def __init__(self, name: str = COEFFICIENT_OF_DISCRIMINATION_NAME):
  """Initializes coefficient of discrimination metric.

  Args:
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(_coefficient_of_discrimination),
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

ConfusionMatrixAtThresholds

ConfusionMatrixAtThresholds(
    thresholds: List[float],
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: Metric

Confusion matrix at thresholds.

Initializes confusion matrix at thresholds.

PARAMETER	DESCRIPTION
thresholds	Thresholds to use for confusion matrix. TYPE: List[float]
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: List[float],
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes confusion matrix at thresholds.

  Args:
    thresholds: Thresholds to use for confusion matrix.
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      thresholds=thresholds,
      name=name,
      top_k=top_k,
      class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

ConfusionMatrixPlot

ConfusionMatrixPlot(
    num_thresholds: int = DEFAULT_NUM_THRESHOLDS,
    name: str = CONFUSION_MATRIX_PLOT_NAME,
    **kwargs,
)

Bases: Metric

Confusion matrix plot.

Initializes confusion matrix plot.

PARAMETER	DESCRIPTION
num_thresholds	Number of thresholds to use when discretizing the curve. Values must be > 1. Defaults to 1000. TYPE: int DEFAULT: DEFAULT_NUM_THRESHOLDS
name	Metric name. TYPE: str DEFAULT: CONFUSION_MATRIX_PLOT_NAME
**kwargs	(Optional) Additional args to pass along to init (and eventually on to _confusion_matrix_plot). These kwargs are useful for subclasses to pass information from their init to the create_computation_fn. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/confusion_matrix_plot.py

def __init__(self,
             num_thresholds: int = DEFAULT_NUM_THRESHOLDS,
             name: str = CONFUSION_MATRIX_PLOT_NAME,
             **kwargs):
  """Initializes confusion matrix plot.

  Args:
    num_thresholds: Number of thresholds to use when discretizing the curve.
      Values must be > 1. Defaults to 1000.
    name: Metric name.
    **kwargs: (Optional) Additional args to pass along to init (and eventually
      on to _confusion_matrix_plot). These kwargs are useful for subclasses to
      pass information from their init to the create_computation_fn.
  """
  super().__init__(
      metric_util.merge_per_key_computations(self._confusion_matrix_plot),
      num_thresholds=num_thresholds,
      name=name,
      **kwargs)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

DerivedMetricComputation

Bases: NamedTuple('DerivedMetricComputation', [('keys', List[MetricKey]), ('result', Callable)])

DerivedMetricComputation derives its result from other computations.

When creating derived metric computations it is recommended (but not required) that the underlying MetricComputations that they depend on are defined at the same time. This is to avoid having to pre-construct and pass around all the required dependencies in order to construct a derived metric. The evaluation pipeline is responsible for de-duplicating overlapping MetricComputations so that only one computation is actually run.

A DerivedMetricComputation is uniquely identified by the combination of the result function's name and the keys. Duplicate computations will be removed automatically.

ATTRIBUTE	DESCRIPTION
keys	List of metric keys associated with derived computation. If the keys are defined as part of the computation then this may be empty in which case only the result function name will be used for identifying computation uniqueness.
result	Function (called per slice) to compute the result using the results of other metric computations.

DiagnosticOddsRatio

DiagnosticOddsRatio(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Diagnostic odds ratio (DOR).

Initializes diagnostic odds ratio.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes diagnostic odds ratio.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  if fn > 0.0 and fp > 0.0 and tn > 0.0:
    return (tp / fn) / (fp / tn)
  else:
    return float('nan')

ExactMatch

ExactMatch(
    name: str = EXACT_MATCH_NAME,
    convert_to: Optional[str] = None,
)

Bases: Metric

Exact Match Metric.

Initializes exact match metric.

PARAMETER	DESCRIPTION
name	The name of the metric to use. TYPE: str DEFAULT: EXACT_MATCH_NAME
convert_to	The conversion to perform before checking equality. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/exact_match.py

def __init__(self,
             name: str = EXACT_MATCH_NAME,
             convert_to: Optional[str] = None):
  """Initializes exact match metric.

  Args:
    name: The name of the metric to use.
    convert_to: The conversion to perform before checking equality.
  """

  super().__init__(
      metric_util.merge_per_key_computations(_exact_match),
      name=name,
      convert_to=convert_to)
  if convert_to and convert_to not in _CONVERT_TO_VALUES:
    raise ValueError('convert_to can only be one of the following: %s' %
                     str(convert_to))

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

ExampleCount

ExampleCount(name: str = EXAMPLE_COUNT_NAME)

Bases: Metric

Example count.

Note that although the example_count is independent of the model, this metric will be associated with a model for consistency with other metrics.

Initializes example count.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: EXAMPLE_COUNT_NAME

Source code in tensorflow_model_analysis/metrics/example_count.py

def __init__(self, name: str = EXAMPLE_COUNT_NAME):
  """Initializes example count.

  Args:
    name: Metric name.
  """

  super().__init__(example_count, name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Always disable confidence intervals for ExampleCount.

Confidence intervals capture uncertainty in a metric if it were computed on more examples. For ExampleCount, this sort of uncertainty is not meaningful, so confidence intervals are disabled.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

F1Score

F1Score(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

F1 score.

Initializes F1 score.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes F1 score.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn
  # This is the harmonic mean of precision and recall or the same as
  # 2 * (precision * recall) / (precision + recall).
  # See https://en.wikipedia.org/wiki/Confusion_matrix for more information.
  return _divide_only_positive_denominator(
      numerator=2 * tp, denominator=2 * tp + fp + fn
  )

FN

FN(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: FalseNegatives

Alias for FalseNegatives.

Initializes FN metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes FN metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return fn

FNR

FNR(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: MissRate

Alias for MissRate.

Initializes FNR metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes FNR metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(fn, fn + tp)

FP

FP(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: FalsePositives

Alias for FalsePositives.

Initializes FP metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes FP metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return fp

FPR

FPR(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: FallOut

Alias for FallOut.

Initializes FPR metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes FPR metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tp, fn
  return _divide_only_positive_denominator(fp, fp + tn)

FallOut

FallOut(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Fall-out (FPR).

Initializes fall-out metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use for fall-out. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes fall-out metric.

  Args:
    thresholds: (Optional) Thresholds to use for fall-out. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tp, fn
  return _divide_only_positive_denominator(fp, fp + tn)

FalseDiscoveryRate

FalseDiscoveryRate(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

False discovery rate (FDR).

Initializes false discovery rate.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes false discovery rate.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fn
  return _divide_only_positive_denominator(fp, fp + tp)

FalseNegatives

FalseNegatives(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Calculates the number of false negatives.

If sample_weight is given, calculates the sum of the weights of false negatives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes FalseNegatives metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) Defaults to [0.5]. A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes FalseNegatives metric.

  Args:
    thresholds: (Optional) Defaults to [0.5]. A float value or a python
      list/tuple of float threshold values in [0, 1]. A threshold is compared
      with prediction values to determine the truth value of predictions
      (i.e., above the threshold is `true`, below is `false`). One metric
      value is generated for each threshold value.
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return fn

FalseOmissionRate

FalseOmissionRate(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

False omission rate (FOR).

Initializes false omission rate.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes false omission rate.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tp, fp
  return _divide_only_positive_denominator(fn, fn + tn)

FalsePositives

FalsePositives(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Calculates the number of false positives.

If sample_weight is given, calculates the sum of the weights of false positives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes FalsePositives metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) Defaults to [0.5]. A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes FalsePositives metric.

  Args:
    thresholds: (Optional) Defaults to [0.5]. A float value or a python
      list/tuple of float threshold values in [0, 1]. A threshold is compared
      with prediction values to determine the truth value of predictions
      (i.e., above the threshold is `true`, below is `false`). One metric
      value is generated for each threshold value.
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return fp

FowlkesMallowsIndex

FowlkesMallowsIndex(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Fowlkes-Mallows index (FM).

Initializes fowlkes-mallows index.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes fowlkes-mallows index.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn
  ppv_denominator = tp + fp
  tpr_denominator = tp + fn
  if ppv_denominator > 0.0 and tpr_denominator > 0.0:
    ppv = tp / ppv_denominator
    tnr = tp / tpr_denominator
    return _pos_sqrt(ppv * tnr)
  else:
    return float('nan')

Informedness

Informedness(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Informedness or bookmaker informedness (BM).

Initializes informedness.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes informedness.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  positives = tp + fn
  negatives = tn + fp
  if positives > 0.0 and negatives > 0.0:
    tpr = tp / positives
    tnr = tn / negatives
    return tpr + tnr - 1
  else:
    return float('nan')

Markedness

Markedness(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Markedness (MK) or deltaP.

Initializes markedness.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes markedness.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  ppv_denominator = tp + fp
  npv_denominator = tn + fn
  if ppv_denominator > 0.0 and npv_denominator > 0.0:
    ppv = tp / ppv_denominator
    npv = tn / npv_denominator
    return ppv + npv - 1
  else:
    return float('nan')

MatthewsCorrelationCoefficient

MatthewsCorrelationCoefficient(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Matthews corrrelation coefficient (MCC).

Initializes matthews corrrelation coefficient.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes matthews corrrelation coefficient.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return _divide_only_positive_denominator(
      numerator=tp * tn - fp * fn,
      denominator=_pos_sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn)),
  )

MaxRecall

MaxRecall(
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
    **kwargs,
)

Bases: Recall

Computes the max recall of the predictions with respect to the labels.

The metric uses true positives and false negatives to compute recall by dividing the true positives by the sum of true positives and false negatives.

Effectively the recall at threshold = epsilon(1.0e-12). It is equilvalent to the recall defined in COCO metrics.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes MaxRecall metrics, it calculates the maximum recall.

PARAMETER	DESCRIPTION
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
**kwargs	(Optional) Additional args to pass along to init (and eventually on to _metric_computation and _metric_value) DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None,
             **kwargs):
  """Initializes MaxRecall metrics, it calculates the maximum recall.

  Args:
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
    **kwargs: (Optional) Additional args to pass along to init (and eventually
      on to _metric_computation and _metric_value)
  """
  super().__init__(
      thresholds=_DEFAULT_THRESHOLD_FOR_MAX_RECALL,
      top_k=top_k,
      class_id=class_id,
      name=name,
      **kwargs)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(tp, tp + fn)

Mean

Mean(
    feature_key_path: _KeyPath,
    example_weights_key_path: Optional[_KeyPath] = None,
    name: Optional[str] = None,
)

Bases: Metric

Mean metric.

Initializes mean metric.

PARAMETER	DESCRIPTION
feature_key_path	key path to feature to calculate the mean of. TYPE: _KeyPath
example_weights_key_path	key path to example weights. TYPE: Optional[_KeyPath] DEFAULT: None
name	Metric base name. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/stats.py

def __init__(
    self,
    feature_key_path: _KeyPath,
    example_weights_key_path: Optional[_KeyPath] = None,
    name: Optional[str] = None,
):
  """Initializes mean metric.

  Args:
    feature_key_path: key path to feature to calculate the mean of.
    example_weights_key_path: key path to example weights.
    name: Metric base name.
  """
  super().__init__(
      _mean_metric,
      feature_key_path=feature_key_path,
      example_weights_key_path=example_weights_key_path,
      name=name or f"{_MEAN_METRIC_BASE_NAME}_{'.'.join(feature_key_path)}",
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanAbsoluteAttributions

MeanAbsoluteAttributions(
    name: str = MEAN_ABSOLUTE_ATTRIBUTIONS_NAME,
)

Bases: AttributionsMetric

Mean aboslute attributions metric.

Initializes mean absolute attributions metric.

PARAMETER	DESCRIPTION
name	Attribution metric name. TYPE: str DEFAULT: MEAN_ABSOLUTE_ATTRIBUTIONS_NAME

Source code in tensorflow_model_analysis/metrics/attributions.py

def __init__(self, name: str = MEAN_ABSOLUTE_ATTRIBUTIONS_NAME):
  """Initializes mean absolute attributions metric.

  Args:
    name: Attribution metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          functools.partial(_mean_attributions, True)),
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanAbsoluteError

MeanAbsoluteError(
    name: str = MEAN_ABSOLUTE_ERROR_NAME, **kwargs
)

Bases: Metric

Calculates the mean of absolute error between labels and predictions.

Formula: error = abs(label - prediction)

The metric computes the mean of absolute error between labels and predictions. The labels and predictions should be floats.

Initializes mean regression error metric.

PARAMETER	DESCRIPTION
name	The name of the metric. TYPE: str DEFAULT: MEAN_ABSOLUTE_ERROR_NAME
**kwargs	Additional named keyword arguments. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/mean_regression_error.py

def __init__(self, name: str = MEAN_ABSOLUTE_ERROR_NAME, **kwargs):
  """Initializes mean regression error metric.

  Args:
    name: The name of the metric.
    **kwargs: Additional named keyword arguments.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _mean_absolute_error_computations
      ),
      name=name,
      **kwargs
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanAbsolutePercentageError

MeanAbsolutePercentageError(
    name: str = MEAN_ABSOLUTE_PERCENTAGE_ERROR_NAME,
    **kwargs,
)

Bases: Metric

Calculates the mean of absolute percentage error.

Formula: error = 100 * abs( (label - prediction) / label )

The metric computes the mean of absolute percentage error between labels and predictions. The labels and predictions should be floats.

Initializes mean regression error metric.

PARAMETER	DESCRIPTION
name	The name of the metric. TYPE: str DEFAULT: MEAN_ABSOLUTE_PERCENTAGE_ERROR_NAME
**kwargs	Additional named keyword arguments. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/mean_regression_error.py

def __init__(self, name: str = MEAN_ABSOLUTE_PERCENTAGE_ERROR_NAME, **kwargs):
  """Initializes mean regression error metric.

  Args:
    name: The name of the metric.
    **kwargs: Additional named keyword arguments.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _mean_absolute_percentage_error_computations
      ),
      name=name,
      **kwargs
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanAttributions

MeanAttributions(name: str = MEAN_ATTRIBUTIONS_NAME)

Bases: AttributionsMetric

Mean attributions metric.

Initializes mean attributions metric.

PARAMETER	DESCRIPTION
name	Attribution metric name. TYPE: str DEFAULT: MEAN_ATTRIBUTIONS_NAME

Source code in tensorflow_model_analysis/metrics/attributions.py

def __init__(self, name: str = MEAN_ATTRIBUTIONS_NAME):
  """Initializes mean attributions metric.

  Args:
    name: Attribution metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          functools.partial(_mean_attributions, False)),
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanLabel

MeanLabel(name: str = MEAN_LABEL_NAME)

Bases: Metric

Mean label.

Initializes mean label.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: MEAN_LABEL_NAME

Source code in tensorflow_model_analysis/metrics/calibration.py

def __init__(self, name: str = MEAN_LABEL_NAME):
  """Initializes mean label.

  Args:
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(_mean_label), name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanPrediction

MeanPrediction(name: str = MEAN_PREDICTION_NAME)

Bases: Metric

Mean prediction.

Initializes mean prediction.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: MEAN_PREDICTION_NAME

Source code in tensorflow_model_analysis/metrics/calibration.py

def __init__(self, name: str = MEAN_PREDICTION_NAME):
  """Initializes mean prediction.

  Args:
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(_mean_prediction), name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanSquaredError

MeanSquaredError(
    name: str = MEAN_SQUARED_ERROR_NAME, **kwargs
)

Bases: Metric

Calculates the mean of squared error between labels and predictions.

Formula: error = L2_norm(label - prediction)**2

The metric computes the mean of squared error (square of L2 norm) between labels and predictions. The labels and predictions could be arrays of arbitrary dimensions. Their dimension should match.

Initializes mean regression error metric.

PARAMETER	DESCRIPTION
name	The name of the metric. TYPE: str DEFAULT: MEAN_SQUARED_ERROR_NAME
**kwargs	Additional named keyword arguments. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/mean_regression_error.py

def __init__(self, name: str = MEAN_SQUARED_ERROR_NAME, **kwargs):
  """Initializes mean regression error metric.

  Args:
    name: The name of the metric.
    **kwargs: Additional named keyword arguments.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _mean_squared_error_computations
      ),
      name=name,
      **kwargs
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MeanSquaredLogarithmicError

MeanSquaredLogarithmicError(
    name: str = MEAN_SQUARED_LOGARITHMIC_ERROR_NAME,
    **kwargs,
)

Bases: Metric

Calculates the mean of squared logarithmic error.

Formula: error = L2_norm(log(label + 1) - log(prediction + 1))**2 Note: log of an array will be elementwise, i.e. log([x1, x2]) = [log(x1), log(x2)]

The metric computes the mean of squared logarithmic error (square of L2 norm) between labels and predictions. The labels and predictions could be arrays of arbitrary dimensions. Their dimension should match.

Initializes mean regression error metric.

PARAMETER	DESCRIPTION
name	The name of the metric. TYPE: str DEFAULT: MEAN_SQUARED_LOGARITHMIC_ERROR_NAME
**kwargs	Additional named keyword arguments. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/mean_regression_error.py

def __init__(self, name: str = MEAN_SQUARED_LOGARITHMIC_ERROR_NAME, **kwargs):
  """Initializes mean regression error metric.

  Args:
    name: The name of the metric.
    **kwargs: Additional named keyword arguments.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _mean_squared_logarithmic_error_computations
      ),
      name=name,
      **kwargs
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

Metric

Metric(
    create_computations_fn: Callable[
        ..., MetricComputations
    ],
    **kwargs,
)

Metric wraps a set of metric computations.

This class exists to provide similarity between tfma.metrics.Metric and tf.keras.metics.Metric.

Calling computations creates the metric computations. The parameters passed to init will be combined with the parameters passed to the computations method. This allows some of the parameters (e.g. model_names, output_names, sub_keys) to be set at the time the computations are created instead of when the metric is defined.

Initializes metric.

PARAMETER	DESCRIPTION
create_computations_fn	Function to create the metrics computations (e.g. mean_label, etc). This function should take the args passed to init as as input along with any of eval_config, schema, model_names, output_names, sub_keys, aggregation_type, or query_key (where needed). TYPE: Callable[..., MetricComputations]
**kwargs	Any additional kwargs to pass to create_computations_fn. These should only contain primitive types or lists/dicts of primitive types. The kwargs passed to computations have precendence over these kwargs. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/metric_types.py

def __init__(
    self, create_computations_fn: Callable[..., MetricComputations], **kwargs
):
  """Initializes metric.

  Args:
    create_computations_fn: Function to create the metrics computations (e.g.
      mean_label, etc). This function should take the args passed to __init__
      as as input along with any of eval_config, schema, model_names,
      output_names, sub_keys, aggregation_type, or query_key (where needed).
    **kwargs: Any additional kwargs to pass to create_computations_fn. These
      should only contain primitive types or lists/dicts of primitive types.
      The kwargs passed to computations have precendence over these kwargs.
  """
  self.create_computations_fn = create_computations_fn
  if 'name' in kwargs:
    if not kwargs['name'] and self._default_name():
      kwargs['name'] = self._default_name()  # pylint: disable=assignment-from-none
    name = kwargs['name']
  else:
    name = None
  self.name = name
  self.kwargs = kwargs
  if hasattr(inspect, 'getfullargspec'):
    self._args = inspect.getfullargspec(self.create_computations_fn).args
  else:
    self._args = inspect.getargspec(self.create_computations_fn).args  # pylint: disable=deprecated-method

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MetricComputation

Bases: NamedTuple('MetricComputation', [('keys', List[MetricKey]), ('preprocessors', List[Preprocessor]), ('combiner', CombineFn)])

MetricComputation represents one or more metric computations.

The preprocessors are called with a PCollection of extracts (or list of extracts if query_key is used) to compute the initial combiner input state which is then passed to the combiner. This needs to be done in two steps because slicing happens between the call to the preprocessors and the combiner and this state may end up in multiple slices so we want the representation to be as efficient as possible. If the preprocessors are None, then StandardMetricInputs will be passed.

A MetricComputation is uniquely identified by the combination of the combiner's name and the keys. Duplicate computations will be removed automatically.

ATTRIBUTE	DESCRIPTION
keys	List of metric keys associated with computation. If the keys are defined as part of the computation then this may be empty in which case only the combiner name will be used for identifying computation uniqueness.
preprocessors	Takes a extracts (or a list of extracts) as input (which typically will contain labels, predictions, example weights, and optionally features) and should return the initial state that the combiner will use as input. The output of a processor should only contain information needed by the combiner.
combiner	Takes preprocessor output as input and outputs a tuple: (slice, metric results). The metric results should be a dict from MetricKey to value (float, int, distribution, ...).

MetricKey

Bases: NamedTuple('MetricKey', [('name', str), ('model_name', str), ('output_name', str), ('sub_key', Optional[SubKey]), ('aggregation_type', Optional[AggregationType]), ('example_weighted', Optional[bool]), ('is_diff', bool)])

A MetricKey uniquely identifies a metric.

ATTRIBUTE	DESCRIPTION
name	Metric name. Names starting with '_' are private and will be filtered from the final results. Names starting with two underscores, '__' are reserved for internal use.
model_name	Optional model name (if multi-model evaluation).
output_name	Optional output name (if multi-output model type).
sub_key	Optional sub key.
aggregation_type	Optional Aggregation type.
example_weighted	Indicates whether this metric was weighted by examples.
is_diff	Optional flag to indicate whether this metrics is a diff metric.

Functions

from_proto staticmethod

from_proto(pb: MetricKey) -> MetricKey

Configures class from proto.

Source code in tensorflow_model_analysis/metrics/metric_types.py

@staticmethod
def from_proto(pb: metrics_for_slice_pb2.MetricKey) -> 'MetricKey':
  """Configures class from proto."""
  example_weighted = None
  if pb.HasField('example_weighted'):
    example_weighted = pb.example_weighted.value
  return MetricKey(
      name=pb.name,
      model_name=pb.model_name,
      output_name=pb.output_name,
      sub_key=SubKey.from_proto(pb.sub_key),
      aggregation_type=AggregationType.from_proto(pb.aggregation_type),
      example_weighted=example_weighted,
      is_diff=pb.is_diff,
  )

make_baseline_key

make_baseline_key(model_name: str) -> MetricKey

Source code in tensorflow_model_analysis/metrics/metric_types.py

def make_baseline_key(self, model_name: str) -> 'MetricKey':
  return self._replace(model_name=model_name, is_diff=False)

make_diff_key

make_diff_key() -> MetricKey

Source code in tensorflow_model_analysis/metrics/metric_types.py

def make_diff_key(self) -> 'MetricKey':
  return self._replace(is_diff=True)

to_proto

to_proto() -> MetricKey

Converts key to proto.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def to_proto(self) -> metrics_for_slice_pb2.MetricKey:
  """Converts key to proto."""
  metric_key = metrics_for_slice_pb2.MetricKey()
  if self.name:
    metric_key.name = self.name
  if self.model_name:
    metric_key.model_name = self.model_name
  if self.output_name:
    metric_key.output_name = self.output_name
  if self.sub_key:
    metric_key.sub_key.CopyFrom(self.sub_key.to_proto())
  if self.aggregation_type:
    metric_key.aggregation_type.CopyFrom(self.aggregation_type.to_proto())
  if self.example_weighted is not None:
    metric_key.example_weighted.value = self.example_weighted
  if self.is_diff:
    metric_key.is_diff = self.is_diff
  return metric_key

MinLabelPosition

MinLabelPosition(
    name=MIN_LABEL_POSITION_NAME,
    label_key: Optional[str] = None,
)

Bases: Metric

Min label position metric.

Calculates the least index in a query which has a positive label. The final returned value is the weighted average over all queries in the evaluation set which have at least one labeled entry. Note, ranking is indexed from one, so the optimal value for this metric is one. If there are no labeled rows in the evaluation set, the final output will be zero.

This is a query/ranking based metric so a query_key must also be provided in the associated metrics spec.

Initializes min label position metric.

PARAMETER	DESCRIPTION
name	Metric name. DEFAULT: MIN_LABEL_POSITION_NAME
label_key	Optional label key to override default label. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/min_label_position.py

def __init__(self,
             name=MIN_LABEL_POSITION_NAME,
             label_key: Optional[str] = None):
  """Initializes min label position metric.

  Args:
    name: Metric name.
    label_key: Optional label key to override default label.
  """
  super().__init__(_min_label_position, name=name, label_key=label_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MissRate

MissRate(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Miss rate (FNR).

Initializes miss rate metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use for miss rate. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes miss rate metric.

  Args:
    thresholds: (Optional) Thresholds to use for miss rate. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(fn, fn + tp)

MultiClassConfusionMatrixAtThresholds

MultiClassConfusionMatrixAtThresholds(
    thresholds: Optional[List[float]] = None,
    name: str = MULTI_CLASS_CONFUSION_MATRIX_AT_THRESHOLDS_NAME,
)

Bases: Metric

Multi-class confusion matrix metrics at thresholds.

Computes weighted example counts for all combinations of actual / (top) predicted classes.

The inputs are assumed to contain a single positive label per example (i.e. only one class can be true at a time) while the predictions are assumed to sum to 1.0.

Initializes multi-class confusion matrix.

PARAMETER	DESCRIPTION
thresholds	Optional thresholds, defaults to 0.5 if not specified. If the top prediction is less than a threshold then the associated example will be assumed to have no prediction associated with it (the predicted_class_id will be set to NO_PREDICTED_CLASS_ID). TYPE: Optional[List[float]] DEFAULT: None
name	Metric name. TYPE: str DEFAULT: MULTI_CLASS_CONFUSION_MATRIX_AT_THRESHOLDS_NAME

Source code in tensorflow_model_analysis/metrics/multi_class_confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[List[float]] = None,
             name: str = MULTI_CLASS_CONFUSION_MATRIX_AT_THRESHOLDS_NAME):
  """Initializes multi-class confusion matrix.

  Args:
    thresholds: Optional thresholds, defaults to 0.5 if not specified. If the
      top prediction is less than a threshold then the associated example will
      be assumed to have no prediction associated with it (the
      predicted_class_id will be set to NO_PREDICTED_CLASS_ID).
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _multi_class_confusion_matrix_at_thresholds),
      thresholds=thresholds,
      name=name)  # pytype: disable=wrong-arg-types

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MultiClassConfusionMatrixPlot

MultiClassConfusionMatrixPlot(
    thresholds: Optional[List[float]] = None,
    num_thresholds: Optional[int] = None,
    name: str = MULTI_CLASS_CONFUSION_MATRIX_PLOT_NAME,
)

Bases: Metric

Multi-class confusion matrix plot.

Computes weighted example counts for all combinations of actual / (top) predicted classes.

The inputs are assumed to contain a single positive label per example (i.e. only one class can be true at a time) while the predictions are assumed to sum to 1.0.

Initializes multi-class confusion matrix.

PARAMETER	DESCRIPTION
thresholds	Optional thresholds. If the top prediction is less than a threshold then the associated example will be assumed to have no prediction associated with it (the predicted_class_id will be set to tfma.metrics.NO_PREDICTED_CLASS_ID). Only one of either thresholds or num_thresholds should be used. If both are unset, then [0.0] will be assumed. TYPE: Optional[List[float]] DEFAULT: None
num_thresholds	Number of thresholds to use. The thresholds will be evenly spaced between 0.0 and 1.0 and inclusive of the boundaries (i.e. to configure the thresholds to [0.0, 0.25, 0.5, 0.75, 1.0], the parameter should be set to 5). Only one of either thresholds or num_thresholds should be used. TYPE: Optional[int] DEFAULT: None
name	Metric name. TYPE: str DEFAULT: MULTI_CLASS_CONFUSION_MATRIX_PLOT_NAME

Source code in tensorflow_model_analysis/metrics/multi_class_confusion_matrix_plot.py

def __init__(self,
             thresholds: Optional[List[float]] = None,
             num_thresholds: Optional[int] = None,
             name: str = MULTI_CLASS_CONFUSION_MATRIX_PLOT_NAME):
  """Initializes multi-class confusion matrix.

  Args:
    thresholds: Optional thresholds. If the top prediction is less than a
      threshold then the associated example will be assumed to have no
      prediction associated with it (the predicted_class_id will be set to
      tfma.metrics.NO_PREDICTED_CLASS_ID). Only one of
      either thresholds or num_thresholds should be used. If both are unset,
      then [0.0] will be assumed.
    num_thresholds: Number of thresholds to use. The thresholds will be evenly
      spaced between 0.0 and 1.0 and inclusive of the boundaries (i.e. to
      configure the thresholds to [0.0, 0.25, 0.5, 0.75, 1.0], the parameter
      should be set to 5). Only one of either thresholds or num_thresholds
      should be used.
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _multi_class_confusion_matrix_plot),
      thresholds=thresholds,
      num_thresholds=num_thresholds,
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

MultiLabelConfusionMatrixPlot

MultiLabelConfusionMatrixPlot(
    thresholds: Optional[List[float]] = None,
    num_thresholds: Optional[int] = None,
    name: str = MULTI_LABEL_CONFUSION_MATRIX_PLOT_NAME,
)

Bases: Metric

Multi-label confusion matrix.

For each actual class (positive label) a confusion matrix is computed for each class based on the associated predicted values such that:

TP = positive_prediction_class_label & positive_prediction TN = negative_prediction_class_label & negative_prediction FP = negative_prediction_class_label & positive_prediction FN = positive_prediction_class_label & negative_prediction

For example, given classes 0, 1 and a given threshold, the following matrices will be computed:

Actual: class_0 Predicted: class_0 TP = is_class_0 & is_class_0 & predict_class_0 TN = is_class_0 & not_class_0 & predict_not_class_0 FN = is_class_0 & is_class_0 & predict_not_class_0 FP = is_class_0 & not_class_0 & predict_class_0 Actual: class_0 Predicted: class_1 TP = is_class_0 & is_class_1 & predict_class_1 TN = is_class_0 & not_class_1 & predict_not_class_1 FN = is_class_0 & is_class_1 & predict_not_class_1 FP = is_class_0 & not_class_1 & predict_class_1 Actual: class_1 Predicted: class_0 TP = is_class_1 & is_class_0 & predict_class_0 TN = is_class_1 & not_class_0 & predict_not_class_0 FN = is_class_1 & is_class_0 & predict_not_class_0 FP = is_class_1 & not_class_0 & predict_class_0 Actual: class_1 Predicted: class_1 TP = is_class_1 & is_class_1 & predict_class_1 TN = is_class_1 & not_class_1 & predict_not_class_1 FN = is_class_1 & is_class_1 & predict_not_class_1 FP = is_class_1 & not_class_1 & predict_class_1

Note that unlike the multi-class confusion matrix, the inputs are assumed to be multi-label whereby the predictions may not necessarily sum to 1.0 and multiple classes can be true as the same time.

Initializes multi-label confusion matrix.

PARAMETER	DESCRIPTION
thresholds	Optional thresholds. Only one of either thresholds or num_thresholds should be used. If both are unset, then [0.5] will be assumed. TYPE: Optional[List[float]] DEFAULT: None
num_thresholds	Number of thresholds to use. The thresholds will be evenly spaced between 0.0 and 1.0 and inclusive of the boundaries (i.e. to configure the thresholds to [0.0, 0.25, 0.5, 0.75, 1.0], the parameter should be set to 5). Only one of either thresholds or num_thresholds should be used. TYPE: Optional[int] DEFAULT: None
name	Metric name. TYPE: str DEFAULT: MULTI_LABEL_CONFUSION_MATRIX_PLOT_NAME

Source code in tensorflow_model_analysis/metrics/multi_label_confusion_matrix_plot.py

def __init__(self,
             thresholds: Optional[List[float]] = None,
             num_thresholds: Optional[int] = None,
             name: str = MULTI_LABEL_CONFUSION_MATRIX_PLOT_NAME):
  """Initializes multi-label confusion matrix.

  Args:
    thresholds: Optional thresholds. Only one of either thresholds or
      num_thresholds should be used. If both are unset, then [0.5] will be
      assumed.
    num_thresholds: Number of thresholds to use. The thresholds will be evenly
      spaced between 0.0 and 1.0 and inclusive of the boundaries (i.e. to
      configure the thresholds to [0.0, 0.25, 0.5, 0.75, 1.0], the parameter
      should be set to 5). Only one of either thresholds or num_thresholds
      should be used.
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _multi_label_confusion_matrix_plot),
      thresholds=thresholds,
      num_thresholds=num_thresholds,
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

NDCG

NDCG(
    gain_key: str,
    top_k_list: Optional[List[int]] = None,
    name: str = NDCG_NAME,
)

Bases: Metric

NDCG (normalized discounted cumulative gain) metric.

Calculates NDCG@k for a given set of top_k values calculated from a list of gains (relevance scores) that are sorted based on the associated predictions. The top_k_list can be passed as part of the NDCG metric config or using tfma.MetricsSpec.binarize.top_k_list if configuring multiple top_k metrics. The gain (relevance score) is determined from the value stored in the 'gain_key' feature. The value of NDCG@k returned is a weighted average of NDCG@k over the set of queries using the example weights.

NDCG@k = (DCG@k for the given rank)/(DCG@k DCG@k = sum_{i=1}^k gain_i/log_2(i+1), where gain_i is the gain (relevance score) of the i^th ranked response, indexed from 1.

This is a query/ranking based metric so a query_key must also be provided in the associated tfma.MetricsSpec.

Initializes NDCG.

PARAMETER	DESCRIPTION
gain_key	Key of feature in features dictionary that holds gain values. TYPE: str
top_k_list	Values for top k. This can also be set using the tfma.MetricsSpec.binarize.top_k_list associated with the metric. TYPE: Optional[List[int]] DEFAULT: None
name	Metric name. TYPE: str DEFAULT: NDCG_NAME

Source code in tensorflow_model_analysis/metrics/ndcg.py

def __init__(
    self,
    gain_key: str,
    top_k_list: Optional[List[int]] = None,
    name: str = NDCG_NAME,
):
  """Initializes NDCG.

  Args:
    gain_key: Key of feature in features dictionary that holds gain values.
    top_k_list: Values for top k. This can also be set using the
      tfma.MetricsSpec.binarize.top_k_list associated with the metric.
    name: Metric name.
  """
  super().__init__(_ndcg, gain_key=gain_key, top_k_list=top_k_list, name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

NPV

NPV(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: NegativePredictiveValue

Alias for NegativePredictiveValue.

Initializes PPV metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes PPV metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tp, fp
  return _divide_only_positive_denominator(tn, tn + fn)

NegativeLikelihoodRatio

NegativeLikelihoodRatio(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Negative likelihood ratio (LR-).

Initializes negative likelihood ratio.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes negative likelihood ratio.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  fnr_denominator = fn + tp
  tnr_denominator = tn + fp
  if fnr_denominator > 0.0 and tnr_denominator > 0.0 and tn > 0.0:
    fnr = fn / fnr_denominator
    tnr = tn / tnr_denominator
    return fnr / tnr
  else:
    return float('nan')

NegativePredictiveValue

NegativePredictiveValue(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Negative predictive value (NPV).

Initializes negative predictive value.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes negative predictive value.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tp, fp
  return _divide_only_positive_denominator(tn, tn + fn)

ObjectDetectionConfusionMatrixPlot

ObjectDetectionConfusionMatrixPlot(
    num_thresholds: int = DEFAULT_NUM_THRESHOLDS,
    name: str = CONFUSION_MATRIX_PLOT_NAME,
    iou_threshold: Optional[float] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: ConfusionMatrixPlot

Object Detection Confusion matrix plot.

Initializes confusion matrix plot for object detection.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
num_thresholds	Number of thresholds to use when discretizing the curve. Values must be > 1. Defaults to 1000. TYPE: int DEFAULT: DEFAULT_NUM_THRESHOLDS
name	Metric name. TYPE: str DEFAULT: CONFUSION_MATRIX_PLOT_NAME
iou_threshold	(Optional) Thresholds for a detection and ground truth pair with specific iou to be considered as a match. Default to 0.5 TYPE: Optional[float] DEFAULT: None
class_id	(Optional) The class id for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class id. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) A tuple (inclusive) representing the area-range for objects to be considered for metrics. Default to (0, inf). TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. Default to None. TYPE: Optional[int] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_confusion_matrix_plot.py

def __init__(self,
             num_thresholds: int = DEFAULT_NUM_THRESHOLDS,
             name: str = CONFUSION_MATRIX_PLOT_NAME,
             iou_threshold: Optional[float] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes confusion matrix plot for object detection.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    num_thresholds: Number of thresholds to use when discretizing the curve.
      Values must be > 1. Defaults to 1000.
    name: Metric name.
    iou_threshold: (Optional) Thresholds for a detection and ground truth pair
      with specific iou to be considered as a match. Default to 0.5
    class_id: (Optional) The class id for calculating metrics.
    class_weight: (Optional) The weight associated with the object class id.
    area_range: (Optional) A tuple (inclusive) representing the area-range for
      objects to be considered for metrics. Default to (0, inf).
    max_num_detections: (Optional) The maximum number of detections for a
      single image. Default to None.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """
  super().__init__(
      num_thresholds=num_thresholds,
      name=name,
      class_id=class_id,
      iou_threshold=iou_threshold,
      area_range=area_range,
      max_num_detections=max_num_detections,
      class_weight=class_weight,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

ObjectDetectionMaxRecall

ObjectDetectionMaxRecall(
    name: Optional[str] = None,
    iou_threshold: Optional[float] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: MaxRecall

Computes the max recall of the predictions with respect to the labels.

The metric uses true positives and false negatives to compute recall by dividing the true positives by the sum of true positives and false negatives.

Effectively the recall at threshold = epsilon(1.0e-12). It is equilvalent to the recall defined in COCO metrics.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes MaxRecall metrics, it calculates the maximum recall.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
iou_threshold	(Optional) Thresholds for a detection and ground truth pair with specific iou to be considered as a match. Default to 0.5 TYPE: Optional[float] DEFAULT: None
class_id	(Optional) The class id for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class id. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) A tuple (inclusive) representing the area-range for objects to be considered for metrics. Default to (0, inf). TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. Default to None. TYPE: Optional[int] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_confusion_matrix_metrics.py

def __init__(self,
             name: Optional[str] = None,
             iou_threshold: Optional[float] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes MaxRecall metrics, it calculates the maximum recall.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    name: (Optional) string name of the metric instance.
    iou_threshold: (Optional) Thresholds for a detection and ground truth pair
      with specific iou to be considered as a match. Default to 0.5
    class_id: (Optional) The class id for calculating metrics.
    class_weight: (Optional) The weight associated with the object class id.
    area_range: (Optional) A tuple (inclusive) representing the area-range for
      objects to be considered for metrics. Default to (0, inf).
    max_num_detections: (Optional) The maximum number of detections for a
      single image. Default to None.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """
  super().__init__(
      name=name,
      class_id=class_id,
      iou_threshold=iou_threshold,
      area_range=area_range,
      max_num_detections=max_num_detections,
      class_weight=class_weight,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(tp, tp + fn)

ObjectDetectionPrecision

ObjectDetectionPrecision(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    iou_threshold: Optional[float] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: Precision

Computes the precision of the predictions with respect to the labels.

The metric uses true positives and false positives to compute precision by dividing the true positives by the sum of true positives and false positives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes Recall metric.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
thresholds	(Optional) A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. The default is to calculate precision with thresholds=0.5. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
iou_threshold	(Optional) Thresholds for a detection and ground truth pair with specific iou to be considered as a match. Default to 0.5 TYPE: Optional[float] DEFAULT: None
class_id	(Optional) The class id for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class id. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) A tuple (inclusive) representing the area-range for objects to be considered for metrics. Default to (0, inf). TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. Default to None. TYPE: Optional[int] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             iou_threshold: Optional[float] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes Recall metric.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    thresholds: (Optional) A float value or a python list/tuple of float
      threshold values in [0, 1]. A threshold is compared with prediction
      values to determine the truth value of predictions (i.e., above the
      threshold is `true`, below is `false`). One metric value is generated
      for each threshold value. The default is to calculate precision with
      `thresholds=0.5`.
    name: (Optional) string name of the metric instance.
    iou_threshold: (Optional) Thresholds for a detection and ground truth pair
      with specific iou to be considered as a match. Default to 0.5
    class_id: (Optional) The class id for calculating metrics.
    class_weight: (Optional) The weight associated with the object class id.
    area_range: (Optional) A tuple (inclusive) representing the area-range for
      objects to be considered for metrics. Default to (0, inf).
    max_num_detections: (Optional) The maximum number of detections for a
      single image. Default to None.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """
  super().__init__(
      thresholds=thresholds,
      name=name,
      class_id=class_id,
      iou_threshold=iou_threshold,
      area_range=area_range,
      max_num_detections=max_num_detections,
      class_weight=class_weight,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fn
  return _divide_only_positive_denominator(tp, tp + fp)

ObjectDetectionPrecisionAtRecall

ObjectDetectionPrecisionAtRecall(
    recall: Union[float, List[float]],
    thresholds: Optional[List[float]] = None,
    num_thresholds: Optional[int] = None,
    name: Optional[str] = None,
    iou_threshold: Optional[float] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: PrecisionAtRecall

Computes best precision where recall is >= specified value.

The threshold for the given recall value is computed and used to evaluate the corresponding precision.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes PrecisionAtRecall metric.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
recall	A scalar or a list of scalar values in range [0, 1]. TYPE: Union[float, List[float]]
thresholds	(Optional) Thresholds to use for calculating the matrices. Use one of either thresholds or num_thresholds. TYPE: Optional[List[float]] DEFAULT: None
num_thresholds	(Optional) Defaults to 1000. The number of thresholds to use for matching the given recall. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
iou_threshold	(Optional) Thresholds for a detection and ground truth pair with specific iou to be considered as a match. Default to 0.5 TYPE: Optional[float] DEFAULT: None
class_id	(Optional) The class id for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class id. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) A tuple (inclusive) representing the area-range for objects to be considered for metrics. Default to (0, inf). TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. Default to None. TYPE: Optional[int] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_confusion_matrix_metrics.py

def __init__(self,
             recall: Union[float, List[float]],
             thresholds: Optional[List[float]] = None,
             num_thresholds: Optional[int] = None,
             name: Optional[str] = None,
             iou_threshold: Optional[float] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes PrecisionAtRecall metric.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    recall: A scalar or a list of scalar values in range `[0, 1]`.
    thresholds: (Optional) Thresholds to use for calculating the matrices. Use
      one of either thresholds or num_thresholds.
    num_thresholds: (Optional) Defaults to 1000. The number of thresholds to
      use for matching the given recall.
    name: (Optional) string name of the metric instance.
    iou_threshold: (Optional) Thresholds for a detection and ground truth pair
      with specific iou to be considered as a match. Default to 0.5
    class_id: (Optional) The class id for calculating metrics.
    class_weight: (Optional) The weight associated with the object class id.
    area_range: (Optional) A tuple (inclusive) representing the area-range for
      objects to be considered for metrics. Default to (0, inf).
    max_num_detections: (Optional) The maximum number of detections for a
      single image. Default to None.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """
  for r in [recall] if isinstance(recall, float) else recall:
    if r < 0 or r > 1:
      raise ValueError('Argument `recall` must be in the range [0, 1]. '
                       f'Received: recall={r}')

  super().__init__(
      thresholds=thresholds,
      num_thresholds=num_thresholds,
      recall=recall,
      name=name,
      class_id=class_id,
      iou_threshold=iou_threshold,
      area_range=area_range,
      max_num_detections=max_num_detections,
      class_weight=class_weight,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

ObjectDetectionRecall

ObjectDetectionRecall(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    iou_threshold: Optional[float] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: Recall

Computes the recall of the predictions with respect to the labels.

The metric uses true positives and false negatives to compute recall by dividing the true positives by the sum of true positives and false negatives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes Recall metric.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
thresholds	(Optional) A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. The default is to calculate recall with thresholds=0.5. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
iou_threshold	(Optional) Thresholds for a detection and ground truth pair with specific iou to be considered as a match. Default to 0.5 TYPE: Optional[float] DEFAULT: None
class_id	(Optional) The class id for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class id. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) A tuple (inclusive) representing the area-range for objects to be considered for metrics. Default to (0, inf). TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. Default to None. TYPE: Optional[int] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             iou_threshold: Optional[float] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes Recall metric.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    thresholds: (Optional) A float value or a python list/tuple of float
      threshold values in [0, 1]. A threshold is compared with prediction
      values to determine the truth value of predictions (i.e., above the
      threshold is `true`, below is `false`). One metric value is generated
      for each threshold value. The default is to calculate recall with
      `thresholds=0.5`.
    name: (Optional) string name of the metric instance.
    iou_threshold: (Optional) Thresholds for a detection and ground truth pair
      with specific iou to be considered as a match. Default to 0.5
    class_id: (Optional) The class id for calculating metrics.
    class_weight: (Optional) The weight associated with the object class id.
    area_range: (Optional) A tuple (inclusive) representing the area-range for
      objects to be considered for metrics. Default to (0, inf).
    max_num_detections: (Optional) The maximum number of detections for a
      single image. Default to None.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """
  super().__init__(
      thresholds=thresholds,
      name=name,
      class_id=class_id,
      iou_threshold=iou_threshold,
      area_range=area_range,
      max_num_detections=max_num_detections,
      class_weight=class_weight,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(tp, tp + fn)

ObjectDetectionThresholdAtRecall

ObjectDetectionThresholdAtRecall(
    recall: Union[float, List[float]],
    thresholds: Optional[List[float]] = None,
    num_thresholds: Optional[int] = None,
    name: Optional[str] = None,
    iou_threshold: Optional[float] = None,
    class_id: Optional[int] = None,
    class_weight: Optional[float] = None,
    area_range: Optional[Tuple[float, float]] = None,
    max_num_detections: Optional[int] = None,
    labels_to_stack: Optional[List[str]] = None,
    predictions_to_stack: Optional[List[str]] = None,
    num_detections_key: Optional[str] = None,
    allow_missing_key: bool = False,
)

Bases: ThresholdAtRecall

Computes maximum threshold where recall is >= specified value.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes ThresholdAtRecall metric.

The metric supports using multiple outputs to form the labels/predictions if the user specifies the label/predcition keys to stack. In this case, the metric is not expected to work with multi-outputs. The metric only supports multi outputs if the output of model is already pre-stacked in the expected format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for predictions.

PARAMETER	DESCRIPTION
recall	A scalar or a list of scalar values in range [0, 1]. TYPE: Union[float, List[float]]
thresholds	(Optional) Thresholds to use for calculating the matrices. Use one of either thresholds or num_thresholds. TYPE: Optional[List[float]] DEFAULT: None
num_thresholds	(Optional) Defaults to 1000. The number of thresholds to use for matching the given recall. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
iou_threshold	(Optional) Thresholds for a detection and ground truth pair with specific iou to be considered as a match. Default to 0.5 TYPE: Optional[float] DEFAULT: None
class_id	(Optional) The class id for calculating metrics. TYPE: Optional[int] DEFAULT: None
class_weight	(Optional) The weight associated with the object class id. TYPE: Optional[float] DEFAULT: None
area_range	(Optional) A tuple (inclusive) representing the area-range for objects to be considered for metrics. Default to (0, inf). TYPE: Optional[Tuple[float, float]] DEFAULT: None
max_num_detections	(Optional) The maximum number of detections for a single image. Default to None. TYPE: Optional[int] DEFAULT: None
labels_to_stack	(Optional) Keys for columns to be stacked as a single numpy array as the labels. It is searched under the key labels, features and transformed features. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] TYPE: Optional[List[str]] DEFAULT: None
predictions_to_stack	(Optional) Output names for columns to be stacked as a single numpy array as the prediction. It should be the model's output names. The desired format is [left bounadary, top boudnary, right boundary, bottom boundary, class id, confidence score]. e.g. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'scores'] TYPE: Optional[List[str]] DEFAULT: None
num_detections_key	(Optional) An output name in which to find the number of detections to use for evaluation for a given example. It does nothing if predictions_to_stack is not set. The value for this output should be a scalar value or a single-value tensor. The stacked predicitions will be truncated with the specified number of detections. TYPE: Optional[str] DEFAULT: None
allow_missing_key	(Optional) If true, the preprocessor will return empty array instead of raising errors. TYPE: bool DEFAULT: False

Source code in tensorflow_model_analysis/metrics/object_detection_confusion_matrix_metrics.py

def __init__(self,
             recall: Union[float, List[float]],
             thresholds: Optional[List[float]] = None,
             num_thresholds: Optional[int] = None,
             name: Optional[str] = None,
             iou_threshold: Optional[float] = None,
             class_id: Optional[int] = None,
             class_weight: Optional[float] = None,
             area_range: Optional[Tuple[float, float]] = None,
             max_num_detections: Optional[int] = None,
             labels_to_stack: Optional[List[str]] = None,
             predictions_to_stack: Optional[List[str]] = None,
             num_detections_key: Optional[str] = None,
             allow_missing_key: bool = False):
  """Initializes ThresholdAtRecall metric.

  The metric supports using multiple outputs to form the labels/predictions if
  the user specifies the label/predcition keys to stack. In this case, the
  metric is not expected to work with multi-outputs. The metric only supports
  multi outputs if the output of model is already pre-stacked in the expected
  format, i.e. ['xmin', 'ymin', 'xmax', 'ymax', 'class_id'] for labels and
  ['xmin', 'ymin', 'xmax', 'ymax', 'class_id', 'confidence scores'] for
  predictions.

  Args:
    recall: A scalar or a list of scalar values in range `[0, 1]`.
    thresholds: (Optional) Thresholds to use for calculating the matrices. Use
      one of either thresholds or num_thresholds.
    num_thresholds: (Optional) Defaults to 1000. The number of thresholds to
      use for matching the given recall.
    name: (Optional) string name of the metric instance.
    iou_threshold: (Optional) Thresholds for a detection and ground truth pair
      with specific iou to be considered as a match. Default to 0.5
    class_id: (Optional) The class id for calculating metrics.
    class_weight: (Optional) The weight associated with the object class id.
    area_range: (Optional) A tuple (inclusive) representing the area-range for
      objects to be considered for metrics. Default to (0, inf).
    max_num_detections: (Optional) The maximum number of detections for a
      single image. Default to None.
    labels_to_stack: (Optional) Keys for columns to be stacked as a single
      numpy array as the labels. It is searched under the key labels, features
      and transformed features. The desired format is [left bounadary, top
      boudnary, right boundary, bottom boundary, class id]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id']
    predictions_to_stack: (Optional) Output names for columns to be stacked as
      a single numpy array as the prediction. It should be the model's output
      names. The desired format is [left bounadary, top boudnary, right
      boundary, bottom boundary, class id, confidence score]. e.g. ['xmin',
      'ymin', 'xmax', 'ymax', 'class_id', 'scores']
    num_detections_key: (Optional) An output name in which to find the number
      of detections to use for evaluation for a given example. It does nothing
      if predictions_to_stack is not set. The value for this output should be
      a scalar value or a single-value tensor. The stacked predicitions will
      be truncated with the specified number of detections.
    allow_missing_key: (Optional) If true, the preprocessor will return empty
      array instead of raising errors.
  """
  for r in [recall] if isinstance(recall, float) else recall:
    if r < 0 or r > 1:
      raise ValueError('Argument `recall` must be in the range [0, 1]. '
                       f'Received: recall={r}')

  super().__init__(
      thresholds=thresholds,
      num_thresholds=num_thresholds,
      recall=recall,
      name=name,
      class_id=class_id,
      iou_threshold=iou_threshold,
      area_range=area_range,
      max_num_detections=max_num_detections,
      class_weight=class_weight,
      labels_to_stack=labels_to_stack,
      predictions_to_stack=predictions_to_stack,
      num_detections_key=num_detections_key,
      allow_missing_key=allow_missing_key)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

PPV

PPV(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: Precision

Alias for Precision.

Initializes PPV metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes PPV metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fn
  return _divide_only_positive_denominator(tp, tp + fp)

PlotKey

Bases: MetricKey

A PlotKey is a metric key that uniquely identifies a plot.

Functions

from_proto staticmethod

from_proto(pb: PlotKey) -> PlotKey

Configures class from proto.

Source code in tensorflow_model_analysis/metrics/metric_types.py

@staticmethod
def from_proto(pb: metrics_for_slice_pb2.PlotKey) -> 'PlotKey':
  """Configures class from proto."""
  example_weighted = None
  if pb.HasField('example_weighted'):
    example_weighted = pb.example_weighted.value
  return PlotKey(
      name=pb.name,
      model_name=pb.model_name,
      output_name=pb.output_name,
      sub_key=SubKey.from_proto(pb.sub_key),
      example_weighted=example_weighted,
  )

make_baseline_key

make_baseline_key(model_name: str) -> MetricKey

Source code in tensorflow_model_analysis/metrics/metric_types.py

def make_baseline_key(self, model_name: str) -> 'MetricKey':
  return self._replace(model_name=model_name, is_diff=False)

make_diff_key

make_diff_key() -> MetricKey

Source code in tensorflow_model_analysis/metrics/metric_types.py

def make_diff_key(self) -> 'MetricKey':
  return self._replace(is_diff=True)

to_proto

to_proto() -> PlotKey

Converts key to proto.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def to_proto(self) -> metrics_for_slice_pb2.PlotKey:  # pytype: disable=signature-mismatch  # overriding-return-type-checks
  """Converts key to proto."""
  plot_key = metrics_for_slice_pb2.PlotKey()
  if self.name:
    plot_key.name = self.name
  if self.model_name:
    plot_key.model_name = self.model_name
  if self.output_name:
    plot_key.output_name = self.output_name
  if self.sub_key:
    plot_key.sub_key.CopyFrom(self.sub_key.to_proto())
  if self.example_weighted is not None:
    plot_key.example_weighted.value = self.example_weighted
  return plot_key

PositiveLikelihoodRatio

PositiveLikelihoodRatio(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Positive likelihood ratio (LR+).

Initializes positive likelihood ratio.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes positive likelihood ratio.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  tpr_denominator = tp + fn
  fpr_denominator = fp + tn
  if tpr_denominator > 0.0 and fpr_denominator > 0.0 and fp > 0.0:
    tpr = tp / tpr_denominator
    fpr = fp / fpr_denominator
    return tpr / fpr
  else:
    return float('nan')

Precision

Precision(
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
    **kwargs,
)

Bases: ConfusionMatrixMetric

Computes the precision of the predictions with respect to the labels.

The metric uses true positives and false positives to compute precision by dividing the true positives by the sum of true positives and false positives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes Precision metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. If neither thresholds nor top_k are set, the default is to calculate precision with thresholds=0.5. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
**kwargs	(Optional) Additional args to pass along to init (and eventually on to _metric_computation and _metric_value). DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None,
             **kwargs):
  """Initializes Precision metric.

  Args:
    thresholds: (Optional) A float value or a python list/tuple of float
      threshold values in [0, 1]. A threshold is compared with prediction
      values to determine the truth value of predictions (i.e., above the
      threshold is `true`, below is `false`). One metric value is generated
      for each threshold value. If neither thresholds nor top_k are set, the
      default is to calculate precision with `thresholds=0.5`.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
    **kwargs: (Optional) Additional args to pass along to init (and eventually
      on to _metric_computation and _metric_value).
  """
  super().__init__(
      thresholds=thresholds,
      top_k=top_k,
      class_id=class_id,
      name=name,
      **kwargs)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fn
  return _divide_only_positive_denominator(tp, tp + fp)

PrecisionAtRecall

PrecisionAtRecall(
    recall: Union[float, List[float]],
    thresholds: Optional[List[float]] = None,
    num_thresholds: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    **kwargs,
)

Bases: ConfusionMatrixMetricBase

Computes best precision where recall is >= specified value.

The threshold for the given recall value is computed and used to evaluate the corresponding precision.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes PrecisionAtRecall metric.

PARAMETER	DESCRIPTION
recall	A scalar or a list of scalar values in range [0, 1]. TYPE: Union[float, List[float]]
thresholds	(Optional) Thresholds to use for calculating the matrices. Use one of either thresholds or num_thresholds. TYPE: Optional[List[float]] DEFAULT: None
num_thresholds	(Optional) Defaults to 1000. The number of thresholds to use for matching the given recall. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
**kwargs	(Optional) Additional args to pass along to init (and eventually on to _metric_computation and _metric_value) DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             recall: Union[float, List[float]],
             thresholds: Optional[List[float]] = None,
             num_thresholds: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             **kwargs):
  """Initializes PrecisionAtRecall metric.


  Args:
    recall: A scalar or a list of scalar values in range `[0, 1]`.
    thresholds: (Optional) Thresholds to use for calculating the matrices. Use
      one of either thresholds or num_thresholds.
    num_thresholds: (Optional) Defaults to 1000. The number of thresholds to
      use for matching the given recall.
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    **kwargs: (Optional) Additional args to pass along to init (and eventually
      on to _metric_computation and _metric_value)
  """
  for r in [recall] if isinstance(recall, float) else recall:
    if r < 0 or r > 1:
      raise ValueError('Argument `recall` must be in the range [0, 1]. '
                       f'Received: recall={r}')

  super().__init__(
      thresholds=thresholds,
      num_thresholds=num_thresholds,
      recall=recall,
      class_id=class_id,
      name=name,
      top_k=top_k,
      **kwargs)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

Preprocessor

Preprocessor(name: Optional[str] = None, **kwargs)

Bases: DoFn

Preprocessor wrapper for preprocessing data in the metric computation.

The preprocessor is a beam.DoFn that takes a extracts (or a list of extracts) as input (which typically will contain labels, predictions, example weights, and optionally features) and should return the initial state that the combiner will use as input. The output of a processor should only contain information needed by the combiner. Note that if a query_key is used the preprocessor will be passed a list of extracts as input representing the extracts that matched the query_key. The special FeaturePreprocessor can be used to add additional features to the default standard metric inputs.

ATTRIBUTE	DESCRIPTION
name	The name of the preprocessor. It should only be accessed by a property function. It is a read only attribute, and is used to distinguish different preprocessors. TYPE: str

Source code in tensorflow_model_analysis/metrics/metric_types.py

def __init__(self, name: Optional[str] = None, **kwargs):
  super().__init__(**kwargs)
  self._name = name

Attributes

name property

name: str

preprocessor_id property

preprocessor_id

Functions

Prevalence

Prevalence(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Prevalence.

Initializes prevalence.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes prevalence.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return _divide_only_positive_denominator(tp + fn, tp + tn + fp + fn)

PrevalenceThreshold

PrevalenceThreshold(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Prevalence threshold (PT).

Initializes prevalence threshold.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes prevalence threshold.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  tpr_denominator = tp + fn
  tnr_denominator = tn + fp
  if tpr_denominator > 0.0 and tnr_denominator > 0.0:
    tpr = tp / tpr_denominator
    tnr = tn / tnr_denominator
    return (_pos_sqrt(tpr * (1 - tnr)) + tnr - 1) / (tpr + tnr - 1)
  else:
    return float('nan')

QueryStatistics

QueryStatistics(
    total_queries_name: str = TOTAL_QUERIES_NAME,
    total_documents_name: str = TOTAL_DOCUMENTS_NAME,
    min_documents_name: str = MIN_DOCUMENTS_NAME,
    max_documents_name: str = MAX_DOCUMENTS_NAME,
)

Bases: Metric

Query statistic metrics.

These metrics are query/ranking based so a query_key must also be provided in the associated metrics spec.

Initializes query statistics metrics.

PARAMETER	DESCRIPTION
total_queries_name	Total queries metric name. TYPE: str DEFAULT: TOTAL_QUERIES_NAME
total_documents_name	Total documents metric name. TYPE: str DEFAULT: TOTAL_DOCUMENTS_NAME
min_documents_name	Min documents name. TYPE: str DEFAULT: MIN_DOCUMENTS_NAME
max_documents_name	Max documents name. TYPE: str DEFAULT: MAX_DOCUMENTS_NAME

Source code in tensorflow_model_analysis/metrics/query_statistics.py

def __init__(self,
             total_queries_name: str = TOTAL_QUERIES_NAME,
             total_documents_name: str = TOTAL_DOCUMENTS_NAME,
             min_documents_name: str = MIN_DOCUMENTS_NAME,
             max_documents_name: str = MAX_DOCUMENTS_NAME):
  """Initializes query statistics metrics.

  Args:
    total_queries_name: Total queries metric name.
    total_documents_name: Total documents metric name.
    min_documents_name: Min documents name.
    max_documents_name: Max documents name.
  """
  super().__init__(
      _query_statistics,
      total_queries_name=total_queries_name,
      total_documents_name=total_documents_name,
      min_documents_name=min_documents_name,
      max_documents_name=max_documents_name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

Recall

Recall(
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
    **kwargs,
)

Bases: ConfusionMatrixMetric

Computes the recall of the predictions with respect to the labels.

The metric uses true positives and false negatives to compute recall by dividing the true positives by the sum of true positives and false negatives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes Recall metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. If neither thresholds nor top_k are set, the default is to calculate precision with thresholds=0.5. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
**kwargs	(Optional) Additional args to pass along to init (and eventually on to _metric_computation and _metric_value) DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None,
             **kwargs):
  """Initializes Recall metric.

  Args:
    thresholds: (Optional) A float value or a python list/tuple of float
      threshold values in [0, 1]. A threshold is compared with prediction
      values to determine the truth value of predictions (i.e., above the
      threshold is `true`, below is `false`). One metric value is generated
      for each threshold value. If neither thresholds nor top_k are set, the
      default is to calculate precision with `thresholds=0.5`.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
    **kwargs: (Optional) Additional args to pass along to init (and eventually
      on to _metric_computation and _metric_value)
  """
  super().__init__(
      thresholds=thresholds,
      top_k=top_k,
      class_id=class_id,
      name=name,
      **kwargs)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(tp, tp + fn)

RecallAtPrecision

RecallAtPrecision(
    precision: float,
    num_thresholds: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
)

Bases: ConfusionMatrixMetricBase

Computes best recall where precision is >= specified value.

For a given score-label-distribution the required precision might not be achievable, in this case 0.0 is returned as recall.

This metric creates three local variables, true_positives, false_positives and false_negatives that are used to compute the recall at the given precision. The threshold for the given precision value is computed and used to evaluate the corresponding recall.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes RecallAtPrecision.

PARAMETER	DESCRIPTION
precision	A scalar value in range [0, 1]. TYPE: float
num_thresholds	(Optional) Defaults to 1000. The number of thresholds to use for matching the given precision. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             precision: float,
             num_thresholds: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None):
  """Initializes RecallAtPrecision.


  Args:
    precision: A scalar value in range `[0, 1]`.
    num_thresholds: (Optional) Defaults to 1000. The number of thresholds to
      use for matching the given precision.
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
  """
  if precision < 0 or precision > 1:
    raise ValueError('Argument `precision` must be in the range [0, 1]. '
                     f'Received: precision={precision}')
  super().__init__(
      num_thresholds=num_thresholds,
      precision=precision,
      class_id=class_id,
      name=name,
      top_k=top_k)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

RelativeCoefficientOfDiscrimination

RelativeCoefficientOfDiscrimination(
    name: str = RELATIVE_COEFFICIENT_OF_DISCRIMINATION_NAME,
)

Bases: Metric

Relative coefficient of discrimination metric.

The relative coefficient of discrimination measures the ratio between the average prediction for the positive examples and the average prediction for the negative examples. This has a very simple intuitive explanation, measuring how much higher is the prediction going to be for a positive example than for a negative example.

Initializes relative coefficient of discrimination metric.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: RELATIVE_COEFFICIENT_OF_DISCRIMINATION_NAME

Source code in tensorflow_model_analysis/metrics/tjur_discrimination.py

def __init__(self, name: str = RELATIVE_COEFFICIENT_OF_DISCRIMINATION_NAME):
  """Initializes relative coefficient of discrimination metric.

  Args:
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          _relative_coefficient_of_discrimination),
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

ScoreDistributionPlot

ScoreDistributionPlot(
    num_thresholds: int = DEFAULT_NUM_THRESHOLDS,
    name: str = SCORE_DISTRIBUTION_PLOT_NAME,
)

Bases: Metric

Score distribution plot.

Initializes confusion matrix plot.

PARAMETER	DESCRIPTION
num_thresholds	Number of thresholds to use when discretizing the curve. Values must be > 1. Defaults to 1000. TYPE: int DEFAULT: DEFAULT_NUM_THRESHOLDS
name	Metric name. TYPE: str DEFAULT: SCORE_DISTRIBUTION_PLOT_NAME

Source code in tensorflow_model_analysis/metrics/score_distribution_plot.py

def __init__(self,
             num_thresholds: int = DEFAULT_NUM_THRESHOLDS,
             name: str = SCORE_DISTRIBUTION_PLOT_NAME):
  """Initializes confusion matrix plot.

  Args:
    num_thresholds: Number of thresholds to use when discretizing the curve.
      Values must be > 1. Defaults to 1000.
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(_confusion_matrix_plot),
      num_thresholds=num_thresholds,
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

SemanticSegmentationConfusionMatrix

SemanticSegmentationConfusionMatrix(
    class_ids: List[int],
    ground_truth_key: str,
    prediction_key: str,
    decode_ground_truth: bool = True,
    decode_prediction: bool = False,
    ignore_ground_truth_id: Optional[int] = None,
    name: Optional[str] = None,
)

Bases: Metric

Computes confusion matrices for semantic segmentation.

Initializes PrecisionAtRecall metric.

PARAMETER	DESCRIPTION
class_ids	the class ids for calculating metrics. TYPE: List[int]
ground_truth_key	the key for storing the ground truth of encoded image with class ids. TYPE: str
prediction_key	the key for storing the predictions of encoded image with class ids. TYPE: str
decode_ground_truth	If true, the ground truth is assumed to be bytes of images and will be decoded. By default it is true assuming the label is the bytes of image. TYPE: bool DEFAULT: True
decode_prediction	If true, the prediction is assumed to be bytes of images and will be decoded. By default it is false assuming the model outputs numpy arrays or tensors. TYPE: bool DEFAULT: False
ignore_ground_truth_id	(Optional) The id of ground truth to be ignored. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/semantic_segmentation_confusion_matrix_metrics.py

def __init__(
    self,
    class_ids: List[int],
    ground_truth_key: str,
    prediction_key: str,
    decode_ground_truth: bool = True,
    decode_prediction: bool = False,
    ignore_ground_truth_id: Optional[int] = None,
    name: Optional[str] = None,
):
  """Initializes PrecisionAtRecall metric.

  Args:
    class_ids: the class ids for calculating metrics.
    ground_truth_key: the key for storing the ground truth of encoded image
      with class ids.
    prediction_key: the key for storing the predictions of encoded image with
      class ids.
    decode_ground_truth: If true, the ground truth is assumed to be bytes of
      images and will be decoded. By default it is true assuming the label is
      the bytes of image.
    decode_prediction: If true, the prediction is assumed to be bytes of
      images and will be decoded. By default it is false assuming the model
      outputs numpy arrays or tensors.
    ignore_ground_truth_id: (Optional) The id of ground truth to be ignored.
    name: (Optional) string name of the metric instance.
  """

  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      name=name,
      class_ids=class_ids,
      ground_truth_key=ground_truth_key,
      prediction_key=prediction_key,
      decode_ground_truth=decode_ground_truth,
      decode_prediction=decode_prediction,
      ignore_ground_truth_id=ignore_ground_truth_id,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

SemanticSegmentationFalsePositive

SemanticSegmentationFalsePositive(
    class_ids: List[int],
    ground_truth_key: str,
    prediction_key: str,
    decode_ground_truth: bool = True,
    decode_prediction: bool = False,
    ignore_ground_truth_id: Optional[int] = None,
    name: Optional[str] = None,
)

Bases: SemanticSegmentationConfusionMatrixMetricBase

Calculates the true postive for semantic segmentation.

Initializes PrecisionAtRecall metric.

PARAMETER	DESCRIPTION
class_ids	the class ids for calculating metrics. TYPE: List[int]
ground_truth_key	the key for storing the ground truth of encoded image with class ids. TYPE: str
prediction_key	the key for storing the predictions of encoded image with class ids. TYPE: str
decode_ground_truth	If true, the ground truth is assumed to be bytes of images and will be decoded. By default it is true assuming the label is the bytes of image. TYPE: bool DEFAULT: True
decode_prediction	If true, the prediction is assumed to be bytes of images and will be decoded. By default it is false assuming the model outputs numpy arrays or tensors. TYPE: bool DEFAULT: False
ignore_ground_truth_id	(Optional) The id of ground truth to be ignored. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/semantic_segmentation_confusion_matrix_metrics.py

def __init__(
    self,
    class_ids: List[int],
    ground_truth_key: str,
    prediction_key: str,
    decode_ground_truth: bool = True,
    decode_prediction: bool = False,
    ignore_ground_truth_id: Optional[int] = None,
    name: Optional[str] = None,
):
  """Initializes PrecisionAtRecall metric.

  Args:
    class_ids: the class ids for calculating metrics.
    ground_truth_key: the key for storing the ground truth of encoded image
      with class ids.
    prediction_key: the key for storing the predictions of encoded image with
      class ids.
    decode_ground_truth: If true, the ground truth is assumed to be bytes of
      images and will be decoded. By default it is true assuming the label is
      the bytes of image.
    decode_prediction: If true, the prediction is assumed to be bytes of
      images and will be decoded. By default it is false assuming the model
      outputs numpy arrays or tensors.
    ignore_ground_truth_id: (Optional) The id of ground truth to be ignored.
    name: (Optional) string name of the metric instance.
  """

  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      name=name,
      class_ids=class_ids,
      ground_truth_key=ground_truth_key,
      prediction_key=prediction_key,
      decode_ground_truth=decode_ground_truth,
      decode_prediction=decode_prediction,
      ignore_ground_truth_id=ignore_ground_truth_id,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

SemanticSegmentationTruePositive

SemanticSegmentationTruePositive(
    class_ids: List[int],
    ground_truth_key: str,
    prediction_key: str,
    decode_ground_truth: bool = True,
    decode_prediction: bool = False,
    ignore_ground_truth_id: Optional[int] = None,
    name: Optional[str] = None,
)

Bases: SemanticSegmentationConfusionMatrixMetricBase

Calculates the true postive for semantic segmentation.

Initializes PrecisionAtRecall metric.

PARAMETER	DESCRIPTION
class_ids	the class ids for calculating metrics. TYPE: List[int]
ground_truth_key	the key for storing the ground truth of encoded image with class ids. TYPE: str
prediction_key	the key for storing the predictions of encoded image with class ids. TYPE: str
decode_ground_truth	If true, the ground truth is assumed to be bytes of images and will be decoded. By default it is true assuming the label is the bytes of image. TYPE: bool DEFAULT: True
decode_prediction	If true, the prediction is assumed to be bytes of images and will be decoded. By default it is false assuming the model outputs numpy arrays or tensors. TYPE: bool DEFAULT: False
ignore_ground_truth_id	(Optional) The id of ground truth to be ignored. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/semantic_segmentation_confusion_matrix_metrics.py

def __init__(
    self,
    class_ids: List[int],
    ground_truth_key: str,
    prediction_key: str,
    decode_ground_truth: bool = True,
    decode_prediction: bool = False,
    ignore_ground_truth_id: Optional[int] = None,
    name: Optional[str] = None,
):
  """Initializes PrecisionAtRecall metric.

  Args:
    class_ids: the class ids for calculating metrics.
    ground_truth_key: the key for storing the ground truth of encoded image
      with class ids.
    prediction_key: the key for storing the predictions of encoded image with
      class ids.
    decode_ground_truth: If true, the ground truth is assumed to be bytes of
      images and will be decoded. By default it is true assuming the label is
      the bytes of image.
    decode_prediction: If true, the prediction is assumed to be bytes of
      images and will be decoded. By default it is false assuming the model
      outputs numpy arrays or tensors.
    ignore_ground_truth_id: (Optional) The id of ground truth to be ignored.
    name: (Optional) string name of the metric instance.
  """

  super().__init__(
      metric_util.merge_per_key_computations(self._metric_computations),
      name=name,
      class_ids=class_ids,
      ground_truth_key=ground_truth_key,
      prediction_key=prediction_key,
      decode_ground_truth=decode_ground_truth,
      decode_prediction=decode_prediction,
      ignore_ground_truth_id=ignore_ground_truth_id,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

SensitivityAtSpecificity

SensitivityAtSpecificity(
    specificity: Union[float, List[float]],
    num_thresholds: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
)

Bases: ConfusionMatrixMetricBase

Computes best sensitivity where specificity is >= specified value.

Sensitivity measures the proportion of actual positives that are correctly identified as such (tp / (tp + fn)). Specificity measures the proportion of actual negatives that are correctly identified as such (tn / (tn + fp)).

The threshold for the given specificity value is computed and used to evaluate the corresponding sensitivity.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

For additional information about specificity and sensitivity, see the following.

Initializes SensitivityAtSpecificity metric.

PARAMETER	DESCRIPTION
specificity	A scalar value in range [0, 1]. TYPE: Union[float, List[float]]
num_thresholds	(Optional) Defaults to 1000. The number of thresholds to use for matching the given specificity. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             specificity: Union[float, List[float]],
             num_thresholds: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None):
  """Initializes SensitivityAtSpecificity metric.


  Args:
    specificity: A scalar value in range `[0, 1]`.
    num_thresholds: (Optional) Defaults to 1000. The number of thresholds to
      use for matching the given specificity.
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
  """
  super().__init__(
      num_thresholds=num_thresholds,
      specificity=specificity,
      class_id=class_id,
      name=name,
      top_k=top_k)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

SetMatchPrecision

SetMatchPrecision(
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    name: Optional[str] = None,
    prediction_class_key: str = "classes",
    prediction_score_key: str = "scores",
    class_key: Optional[str] = None,
    weight_key: Optional[str] = None,
    **kwargs,
)

Bases: Precision

Computes precision for sets of labels and predictions.

The metric deals with labels and predictions which are provided in the format of sets (stored as variable length numpy arrays). The precision is the micro averaged classification precision. The metric is suitable for the case where the number of classes is large or the list of classes could not be provided in advance.

Example: Label: ['cats'], Predictions: {'classes': ['cats, dogs']}

The precision is 0.5.

Initializes Precision metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. If neither thresholds nor top_k are set, the default is to calculate precision with thresholds=0.5. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are truncated and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. When top_k is used, the default threshold is float('-inf'). In this case, unmatched labels are still considered false negative, since they have prediction with confidence score float('-inf'), TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
prediction_class_key	the key name of the classes in prediction. TYPE: str DEFAULT: 'classes'
prediction_score_key	the key name of the scores in prediction. TYPE: str DEFAULT: 'scores'
class_key	(Optional) The key name of the classes in class-weight pairs. If it is not provided, the classes are assumed to be the label classes. TYPE: Optional[str] DEFAULT: None
weight_key	(Optional) The key name of the weights of classes in class-weight pairs. The value in this key should be a numpy array of the same length as the classes in class_key. The key should be stored under the features key. TYPE: Optional[str] DEFAULT: None
**kwargs	(Optional) Additional args to pass along to init (and eventually on to _metric_computations and _metric_values). The args are passed to the precision metric, the confusion matrix metric and binary classification metric. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/set_match_confusion_matrix_metrics.py

def __init__(
    self,
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    name: Optional[str] = None,
    prediction_class_key: str = 'classes',
    prediction_score_key: str = 'scores',
    class_key: Optional[str] = None,
    weight_key: Optional[str] = None,
    **kwargs,
):
  """Initializes Precision metric.

  Args:
    thresholds: (Optional) A float value or a python list/tuple of float
      threshold values in [0, 1]. A threshold is compared with prediction
      values to determine the truth value of predictions (i.e., above the
      threshold is `true`, below is `false`). One metric value is generated
      for each threshold value. If neither thresholds nor top_k are set, the
      default is to calculate precision with `thresholds=0.5`.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are truncated and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. When
      top_k is used, the default threshold is float('-inf'). In this case,
      unmatched labels are still considered false negative, since they have
      prediction with confidence score float('-inf'),
    name: (Optional) string name of the metric instance.
    prediction_class_key: the key name of the classes in prediction.
    prediction_score_key: the key name of the scores in prediction.
    class_key: (Optional) The key name of the classes in class-weight pairs.
      If it is not provided, the classes are assumed to be the label classes.
    weight_key: (Optional) The key name of the weights of classes in
      class-weight pairs. The value in this key should be a numpy array of the
      same length as the classes in class_key. The key should be stored under
      the features key.
    **kwargs: (Optional) Additional args to pass along to init (and eventually
      on to _metric_computations and _metric_values). The args are passed to
      the precision metric, the confusion matrix metric and binary
      classification metric.
  """

  super().__init__(
      thresholds=thresholds,
      top_k=top_k,
      name=name,
      prediction_class_key=prediction_class_key,
      prediction_score_key=prediction_score_key,
      class_key=class_key,
      weight_key=weight_key,
      **kwargs,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fn
  return _divide_only_positive_denominator(tp, tp + fp)

SetMatchRecall

SetMatchRecall(
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    name: Optional[str] = None,
    prediction_class_key: str = "classes",
    prediction_score_key: str = "scores",
    class_key: Optional[str] = None,
    weight_key: Optional[str] = None,
    **kwargs,
)

Bases: Recall

Computes recall for sets of labels and predictions.

The metric deals with labels and predictions which are provided in the format of sets (stored as variable length numpy arrays). The recall is the micro averaged classification recall. The metric is suitable for the case where the number of classes is large or the list of classes could not be provided in advance.

Example: Label: ['cats'], Predictions: {'classes': ['cats, dogs']}

The recall is 1.

Initializes recall metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. If neither thresholds nor top_k are set, the default is to calculate precision with thresholds=0.5. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are truncated and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. When top_k is used, the default threshold is float('-inf'). In this case, unmatched labels are still considered false negative, since they have prediction with confidence score float('-inf'), TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
prediction_class_key	the key name of the classes in prediction. TYPE: str DEFAULT: 'classes'
prediction_score_key	the key name of the scores in prediction. TYPE: str DEFAULT: 'scores'
class_key	(Optional) The key name of the classes in class-weight pairs. If it is not provided, the classes are assumed to be the label classes. TYPE: Optional[str] DEFAULT: None
weight_key	(Optional) The key name of the weights of classes in class-weight pairs. The value in this key should be a numpy array of the same length as the classes in class_key. The key should be stored under the features key. TYPE: Optional[str] DEFAULT: None
**kwargs	(Optional) Additional args to pass along to init (and eventually on to _metric_computations and _metric_values). The args are passed to the recall metric, the confusion matrix metric and binary classification metric. DEFAULT: {}

Source code in tensorflow_model_analysis/metrics/set_match_confusion_matrix_metrics.py

def __init__(
    self,
    thresholds: Optional[Union[float, List[float]]] = None,
    top_k: Optional[int] = None,
    name: Optional[str] = None,
    prediction_class_key: str = 'classes',
    prediction_score_key: str = 'scores',
    class_key: Optional[str] = None,
    weight_key: Optional[str] = None,
    **kwargs,
):
  """Initializes recall metric.

  Args:
    thresholds: (Optional) A float value or a python list/tuple of float
      threshold values in [0, 1]. A threshold is compared with prediction
      values to determine the truth value of predictions (i.e., above the
      threshold is `true`, below is `false`). One metric value is generated
      for each threshold value. If neither thresholds nor top_k are set, the
      default is to calculate precision with `thresholds=0.5`.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are truncated and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. When
      top_k is used, the default threshold is float('-inf'). In this case,
      unmatched labels are still considered false negative, since they have
      prediction with confidence score float('-inf'),
    name: (Optional) string name of the metric instance.
    prediction_class_key: the key name of the classes in prediction.
    prediction_score_key: the key name of the scores in prediction.
    class_key: (Optional) The key name of the classes in class-weight pairs.
      If it is not provided, the classes are assumed to be the label classes.
    weight_key: (Optional) The key name of the weights of classes in
      class-weight pairs. The value in this key should be a numpy array of the
      same length as the classes in class_key. The key should be stored under
      the features key.
    **kwargs: (Optional) Additional args to pass along to init (and eventually
      on to _metric_computations and _metric_values). The args are passed to
      the recall metric, the confusion matrix metric and binary classification
      metric.
  """

  super().__init__(
      thresholds=thresholds,
      top_k=top_k,
      name=name,
      prediction_class_key=prediction_class_key,
      prediction_score_key=prediction_score_key,
      class_key=class_key,
      weight_key=weight_key,
      **kwargs,
  )

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(tp, tp + fn)

Specificity

Specificity(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Specificity (TNR) or selectivity.

Initializes specificity metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use for specificity. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes specificity metric.

  Args:
    thresholds: (Optional) Thresholds to use for specificity. Defaults to
      [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tp, fn
  return _divide_only_positive_denominator(tn, tn + fp)

SpecificityAtSensitivity

SpecificityAtSensitivity(
    sensitivity: float,
    num_thresholds: Optional[int] = None,
    class_id: Optional[int] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
)

Bases: ConfusionMatrixMetricBase

Computes best specificity where sensitivity is >= specified value.

Sensitivity measures the proportion of actual positives that are correctly identified as such (tp / (tp + fn)). Specificity measures the proportion of actual negatives that are correctly identified as such (tn / (tn + fp)).

The threshold for the given sensitivity value is computed and used to evaluate the corresponding specificity.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

For additional information about specificity and sensitivity, see the following.

Initializes SpecificityAtSensitivity metric.

PARAMETER	DESCRIPTION
sensitivity	A scalar value or a list of scalar value in range [0, 1]. TYPE: float
num_thresholds	(Optional) Defaults to 1000. The number of thresholds to use for matching the given sensitivity. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None
name	(Optional) string name of the metric instance. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             sensitivity: float,
             num_thresholds: Optional[int] = None,
             class_id: Optional[int] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None):
  """Initializes SpecificityAtSensitivity metric.


  Args:
    sensitivity: A scalar value or a list of scalar value in range `[0, 1]`.
    num_thresholds: (Optional) Defaults to 1000. The number of thresholds to
      use for matching the given sensitivity.
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
    name: (Optional) string name of the metric instance.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
  """
  super().__init__(
      num_thresholds=num_thresholds,
      sensitivity=sensitivity,
      class_id=class_id,
      name=name,
      top_k=top_k)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

SquaredPearsonCorrelation

SquaredPearsonCorrelation(
    name: str = SQUARED_PEARSON_CORRELATION_NAME,
)

Bases: Metric

Squared pearson correlation (r^2) metric.

Initializes squared pearson correlation (r^2) metric.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: SQUARED_PEARSON_CORRELATION_NAME

Source code in tensorflow_model_analysis/metrics/squared_pearson_correlation.py

def __init__(self, name: str = SQUARED_PEARSON_CORRELATION_NAME):
  """Initializes squared pearson correlation (r^2) metric.

  Args:
    name: Metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(_squared_pearson_correlation),
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

StandardMetricInputs

StandardMetricInputs(
    extracts: Optional[Extracts] = None, **kwargs
)

Bases: StandardExtracts

Standard inputs used by most metric computations.

StandardMetricInputs is a wrapper around Extracts where only the extracts keys used by one or more ExtractsPreprocessors will be present.

Initializes StandardExtracts.

PARAMETER	DESCRIPTION
extracts	Reference to existing extracts to use. TYPE: Optional[Extracts] DEFAULT: None
**kwargs	Name/value pairs to create new extracts from. Only one of either extracts or kwargs should be used. DEFAULT: {}

Source code in tensorflow_model_analysis/utils/util.py

def __init__(self, extracts: Optional[types.Extracts] = None, **kwargs):
  """Initializes StandardExtracts.

  Args:
    extracts: Reference to existing extracts to use.
    **kwargs: Name/value pairs to create new extracts from. Only one of either
      extracts or kwargs should be used.
  """
  if extracts is not None and kwargs:
    raise ValueError('only one of extracts or kwargs should be used')
  if extracts is not None:
    self.extracts = extracts
  else:
    self.extracts = kwargs

Attributes

attributions property

attributions: Optional[DictOfTensorValueMaybeDict]

combined_features property

combined_features: Mapping[str, Any]

example_weight property

example_weight: Optional[TensorValueMaybeMultiLevelDict]

Same as example_weights (DEPRECATED - use example_weights).

example_weights property

example_weights: Optional[TensorValueMaybeMultiLevelDict]

extracts instance-attribute

extracts = extracts

features property

features: Optional[DictOfTensorValueMaybeDict]

inputs property

inputs: Any

label property

label: Optional[TensorValueMaybeMultiLevelDict]

Same as labels (DEPRECATED - use labels).

labels property

labels: Optional[TensorValueMaybeMultiLevelDict]

prediction property

prediction: Optional[TensorValueMaybeMultiLevelDict]

Same as predictions (DEPRECATED - use predictions).

predictions property

predictions: Optional[TensorValueMaybeMultiLevelDict]

transformed_features property

transformed_features: Optional[DictOfTensorValueMaybeDict]

Functions

get_attributions

get_attributions(
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Optional[DictOfTensorValueMaybeDict]

Returns tfma.ATTRIBUTIONS_KEY extract.

Source code in tensorflow_model_analysis/utils/util.py

def get_attributions(
    self,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None
) -> Optional[types.DictOfTensorValueMaybeDict]:
  """Returns tfma.ATTRIBUTIONS_KEY extract."""
  return self.get_by_key(constants.ATTRIBUTIONS_KEY, model_name, output_name)

get_by_key

get_by_key(
    key: str,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Any

Returns item for key possibly filtered by model and/or output names.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_by_key(
    self,
    key: str,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Any:
  if key not in self and key.endswith('s'):
    # The previous version of StandardMetricInputs was a NamedTuple that
    # used label, prediction, and example_weight as the field names. Some
    # tests may be creating StandardMetricInputs using these names, so also
    # search under the non-pluralized form of the key.
    key = key[:-1]
  return super().get_by_key(key, model_name, output_name)

get_combined_features

get_combined_features(
    model_name: Optional[str] = None,
) -> Mapping[str, Any]

Returns a combined extract of transformed features and features.

In case of name collision, transformed features is looked up first and the value is returned when found.

PARAMETER	DESCRIPTION
model_name	Optionally, the model name assosicated to the transformed feature. This has no effect on the raw features extract. TYPE: Optional[str] DEFAULT: None

Source code in tensorflow_model_analysis/utils/util.py

def get_combined_features(self,
                          model_name: Optional[str] = None
                         ) -> Mapping[str, Any]:
  """Returns a combined extract of transformed features and features.

  In case of name collision, transformed features is looked up first and
  the value is returned when found.

  Args:
    model_name: Optionally, the model name assosicated to the transformed
      feature. This has no effect on the raw features extract.
  """
  return collections.ChainMap(
      self.get_transformed_features(model_name) or {},
      self.get_features() or {})

get_example_weights

get_example_weights(
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Optional[TensorValueMaybeMultiLevelDict]

Returns tfma.EXAMPLE_WEIGHTS_KEY extract.

Source code in tensorflow_model_analysis/utils/util.py

def get_example_weights(
    self,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None
) -> Optional[types.TensorValueMaybeMultiLevelDict]:
  """Returns tfma.EXAMPLE_WEIGHTS_KEY extract."""
  return self.get_by_key(constants.EXAMPLE_WEIGHTS_KEY, model_name,
                         output_name)

get_features

get_features() -> Optional[DictOfTensorValueMaybeDict]

Returns tfma.FEATURES_KEY extract.

Source code in tensorflow_model_analysis/utils/util.py

def get_features(self) -> Optional[types.DictOfTensorValueMaybeDict]:
  """Returns tfma.FEATURES_KEY extract."""
  return self.get_by_key(constants.FEATURES_KEY)

get_inputs

get_inputs() -> Any

Returns tfma.INPUT_KEY extract.

Source code in tensorflow_model_analysis/utils/util.py

def get_inputs(self) -> Any:
  """Returns tfma.INPUT_KEY extract."""
  return self[constants.INPUT_KEY]

get_labels

get_labels(
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Optional[TensorValueMaybeMultiLevelDict]

Returns tfma.LABELS_KEY extract.

Source code in tensorflow_model_analysis/utils/util.py

def get_labels(
    self,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None
) -> Optional[types.TensorValueMaybeMultiLevelDict]:
  """Returns tfma.LABELS_KEY extract."""
  return self.get_by_key(constants.LABELS_KEY, model_name, output_name)

get_model_and_output_names

get_model_and_output_names(
    eval_config: EvalConfig,
) -> List[Tuple[Optional[str], Optional[str]]]

Returns a list of model_name-output_name tuples present in extracts.

Source code in tensorflow_model_analysis/utils/util.py

def get_model_and_output_names(
    self, eval_config: config_pb2.EvalConfig
) -> List[Tuple[Optional[str], Optional[str]]]:
  """Returns a list of model_name-output_name tuples present in extracts."""
  if constants.PREDICTIONS_KEY not in self:
    logging.warning('Attempting to get model names and output names from '
                    'extracts that don\'t contain predictions')
    return []
  keys = []
  predictions = self[constants.PREDICTIONS_KEY]
  config_model_names = {spec.name for spec in eval_config.model_specs}
  if not config_model_names:
    config_model_names = {''}
  if isinstance(predictions, Mapping):
    for key in _get_keys(predictions, max_depth=2):
      if len(key) == 1:
        # Because of the dynamic structure of Extracts, we don't know whether
        # a single key is a model name or output name. To distinguish between
        # these cases, we inspect the config.
        if key[0] in config_model_names:
          # multi-model, single output case
          key = key + (None,)
        else:
          # single model, multi-output case
          key = (None,) + key
      keys.append(key)
  else:
    keys = [(None, None)]
  return keys

get_predictions

get_predictions(
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Optional[TensorValueMaybeMultiLevelDict]

Returns tfma.PREDICTIONS_KEY extract.

Source code in tensorflow_model_analysis/utils/util.py

def get_predictions(
    self,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None
) -> Optional[types.TensorValueMaybeMultiLevelDict]:
  """Returns tfma.PREDICTIONS_KEY extract."""
  return self.get_by_key(constants.PREDICTIONS_KEY, model_name, output_name)

get_transformed_features

get_transformed_features(
    model_name: Optional[str] = None,
) -> Optional[DictOfTensorValueMaybeDict]

Returns tfma.TRANSFORMED_FEATURES_KEY extract.

Source code in tensorflow_model_analysis/utils/util.py

def get_transformed_features(
    self,
    model_name: Optional[str] = None
) -> Optional[types.DictOfTensorValueMaybeDict]:
  """Returns tfma.TRANSFORMED_FEATURES_KEY extract."""
  return self.get_by_key(constants.TRANSFORMED_FEATURES_KEY, model_name)

set_labels

set_labels(
    labels: TensorValueMaybeMultiLevelDict,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Optional[TensorValueMaybeMultiLevelDict]

Sets tfma.LABELS_KEY extract for a given model and output.

Source code in tensorflow_model_analysis/utils/util.py

def set_labels(
    self,
    labels: types.TensorValueMaybeMultiLevelDict,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None
) -> Optional[types.TensorValueMaybeMultiLevelDict]:
  """Sets tfma.LABELS_KEY extract for a given model and output."""
  self._set_by_key(
      key=constants.LABELS_KEY,
      value=labels,
      model_name=model_name,
      output_name=output_name)

set_predictions

set_predictions(
    predictions: TensorValueMaybeMultiLevelDict,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None,
) -> Optional[TensorValueMaybeMultiLevelDict]

Sets tfma.PREDICTIONS_KEY extract for a given model and output.

Source code in tensorflow_model_analysis/utils/util.py

def set_predictions(
    self,
    predictions: types.TensorValueMaybeMultiLevelDict,
    model_name: Optional[str] = None,
    output_name: Optional[str] = None
) -> Optional[types.TensorValueMaybeMultiLevelDict]:
  """Sets tfma.PREDICTIONS_KEY extract for a given model and output."""
  self._set_by_key(
      key=constants.PREDICTIONS_KEY,
      value=predictions,
      model_name=model_name,
      output_name=output_name)

SubKey

Bases: NamedTuple('SubKey', [('class_id', int), ('k', int), ('top_k', int)])

A SubKey identifies a sub-types of metrics and plots.

Only one of class_id, k, or top_k can be set at a time.

ATTRIBUTE	DESCRIPTION
class_id	Used with multi-class metrics to identify a specific class ID.
k	Used with multi-class metrics to identify the kth predicted value.
top_k	Used with multi-class and ranking metrics to identify top-k predicted values.

Functions

from_proto staticmethod

from_proto(pb: SubKey) -> Optional[SubKey]

Creates class from proto.

Source code in tensorflow_model_analysis/metrics/metric_types.py

@staticmethod
def from_proto(pb: metrics_for_slice_pb2.SubKey) -> Optional['SubKey']:
  """Creates class from proto."""
  class_id = None
  if pb.HasField('class_id'):
    class_id = pb.class_id.value
  k = None
  if pb.HasField('k'):
    k = pb.k.value
  top_k = None
  if pb.HasField('top_k'):
    top_k = pb.top_k.value
  if class_id is None and k is None and top_k is None:
    return None
  else:
    return SubKey(class_id=class_id, k=k, top_k=top_k)

to_proto

to_proto() -> SubKey

Converts key to proto.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def to_proto(self) -> metrics_for_slice_pb2.SubKey:
  """Converts key to proto."""
  sub_key = metrics_for_slice_pb2.SubKey()
  if self.class_id is not None:
    sub_key.class_id.value = self.class_id
  if self.k is not None:
    sub_key.k.value = self.k
  if self.top_k is not None:
    sub_key.top_k.value = self.top_k
  return sub_key

SymmetricPredictionDifference

SymmetricPredictionDifference(
    name: str = SYMMETRIC_PREDICITON_DIFFERENCE_NAME,
)

Bases: Metric

PredictionDifference computes the avg pointwise diff between models.

Initializes PredictionDifference metric.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: SYMMETRIC_PREDICITON_DIFFERENCE_NAME

Source code in tensorflow_model_analysis/metrics/prediction_difference_metrics.py

def __init__(self, name: str = SYMMETRIC_PREDICITON_DIFFERENCE_NAME):
  """Initializes PredictionDifference metric.

  Args:
    name: Metric name.
  """

  super().__init__(_symmetric_prediction_difference_computations, name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

TN

TN(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: TrueNegatives

Alias for TrueNegatives.

Initializes TN metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes TN metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return tn

TNR

TNR(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: Specificity

Alias for Specificity.

Initializes TNR metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes TNR metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tp, fn
  return _divide_only_positive_denominator(tn, tn + fp)

TP

TP(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: TruePositives

Alias for TruePositives.

Initializes TP metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes TP metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return tp

TPR

TPR(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: Recall

Alias for Recall.

Initializes TPR metric.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes TPR metric."""
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn, fp
  return _divide_only_positive_denominator(tp, tp + fn)

ThreatScore

ThreatScore(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Threat score or critical success index (TS or CSI).

Initializes threat score.

PARAMETER	DESCRIPTION
thresholds	(Optional) Thresholds to use. Defaults to [0.5]. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes threat score.

  Args:
    thresholds: (Optional) Thresholds to use. Defaults to [0.5].
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  del tn
  return _divide_only_positive_denominator(tp, tp + fn + fp)

TotalAbsoluteAttributions

TotalAbsoluteAttributions(
    name: str = TOTAL_ABSOLUTE_ATTRIBUTIONS_NAME,
)

Bases: AttributionsMetric

Total absolute attributions metric.

Initializes total absolute attributions metric.

PARAMETER	DESCRIPTION
name	Attribution metric name. TYPE: str DEFAULT: TOTAL_ABSOLUTE_ATTRIBUTIONS_NAME

Source code in tensorflow_model_analysis/metrics/attributions.py

def __init__(self, name: str = TOTAL_ABSOLUTE_ATTRIBUTIONS_NAME):
  """Initializes total absolute attributions metric.

  Args:
    name: Attribution metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          functools.partial(_total_attributions, True)),
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

TotalAttributions

TotalAttributions(name: str = TOTAL_ATTRIBUTIONS_NAME)

Bases: AttributionsMetric

Total attributions metric.

Initializes total attributions metric.

PARAMETER	DESCRIPTION
name	Attribution metric name. TYPE: str DEFAULT: TOTAL_ATTRIBUTIONS_NAME

Source code in tensorflow_model_analysis/metrics/attributions.py

def __init__(self, name: str = TOTAL_ATTRIBUTIONS_NAME):
  """Initializes total attributions metric.

  Args:
    name: Attribution metric name.
  """
  super().__init__(
      metric_util.merge_per_key_computations(
          functools.partial(_total_attributions, False)),
      name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

TrueNegatives

TrueNegatives(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Calculates the number of true negatives.

If sample_weight is given, calculates the sum of the weights of true negatives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes TrueNegatives metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) Defaults to [0.5]. A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes TrueNegatives metric.

  Args:
    thresholds: (Optional) Defaults to [0.5]. A float value or a python
      list/tuple of float threshold values in [0, 1]. A threshold is compared
      with prediction values to determine the truth value of predictions
      (i.e., above the threshold is `true`, below is `false`). One metric
      value is generated for each threshold value.
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return tn

TruePositives

TruePositives(
    thresholds: Optional[Union[float, List[float]]] = None,
    name: Optional[str] = None,
    top_k: Optional[int] = None,
    class_id: Optional[int] = None,
)

Bases: ConfusionMatrixMetric

Calculates the number of true positives.

If sample_weight is given, calculates the sum of the weights of true positives. This metric creates one local variable, true_positives that is used to keep track of the number of true positives.

If sample_weight is None, weights default to 1. Use sample_weight of 0 to mask values.

Initializes TruePositives metric.

PARAMETER	DESCRIPTION
thresholds	(Optional) Defaults to [0.5]. A float value or a python list/tuple of float threshold values in [0, 1]. A threshold is compared with prediction values to determine the truth value of predictions (i.e., above the threshold is true, below is false). One metric value is generated for each threshold value. TYPE: Optional[Union[float, List[float]]] DEFAULT: None
name	(Optional) Metric name. TYPE: Optional[str] DEFAULT: None
top_k	(Optional) Used with a multi-class model to specify that the top-k values should be used to compute the confusion matrix. The net effect is that the non-top-k values are set to -inf and the matrix is then constructed from the average TP, FP, TN, FN across the classes. When top_k is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. When top_k is set, the default thresholds are [float('-inf')]. TYPE: Optional[int] DEFAULT: None
class_id	(Optional) Used with a multi-class model to specify which class to compute the confusion matrix for. When class_id is used, metrics_specs.binarize settings must not be present. Only one of class_id or top_k should be configured. TYPE: Optional[int] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def __init__(self,
             thresholds: Optional[Union[float, List[float]]] = None,
             name: Optional[str] = None,
             top_k: Optional[int] = None,
             class_id: Optional[int] = None):
  """Initializes TruePositives metric.

  Args:
    thresholds: (Optional) Defaults to [0.5]. A float value or a python
      list/tuple of float threshold values in [0, 1]. A threshold is compared
      with prediction values to determine the truth value of predictions
      (i.e., above the threshold is `true`, below is `false`). One metric
      value is generated for each threshold value.
    name: (Optional) Metric name.
    top_k: (Optional) Used with a multi-class model to specify that the top-k
      values should be used to compute the confusion matrix. The net effect is
      that the non-top-k values are set to -inf and the matrix is then
      constructed from the average TP, FP, TN, FN across the classes. When
      top_k is used, metrics_specs.binarize settings must not be present. Only
      one of class_id or top_k should be configured. When top_k is set, the
      default thresholds are [float('-inf')].
    class_id: (Optional) Used with a multi-class model to specify which class
      to compute the confusion matrix for. When class_id is used,
      metrics_specs.binarize settings must not be present. Only one of
      class_id or top_k should be configured.
  """
  super().__init__(
      thresholds=thresholds, name=name, top_k=top_k, class_id=class_id)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Whether to compute confidence intervals for this metric.

Note that this may not completely remove the computational overhead involved in computing a given metric. This is only respected by the jackknife confidence interval method.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals for this metric.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  # Not all subclasses of ConfusionMatrixMetric support all the __init__
  # parameters as part of their __init__, to avoid deserialization issues
  # where an unsupported parameter is passed to the subclass, filter out any
  # parameters that are None.
  kwargs = copy.copy(self.kwargs)
  for arg in ('thresholds', 'num_thresholds', 'top_k', 'class_id'):
    if kwargs[arg] is None:
      del kwargs[arg]
  return kwargs

result

result(tp: float, tn: float, fp: float, fn: float) -> float

Function for computing metric value from TP, TN, FP, FN values.

Source code in tensorflow_model_analysis/metrics/confusion_matrix_metrics.py

def result(self, tp: float, tn: float, fp: float, fn: float) -> float:
  return tp

WeightedExampleCount

WeightedExampleCount(
    name: str = WEIGHTED_EXAMPLE_COUNT_NAME,
)

Bases: ExampleCount

Weighted example count (deprecated - use ExampleCount).

Initializes weighted example count.

PARAMETER	DESCRIPTION
name	Metric name. TYPE: str DEFAULT: WEIGHTED_EXAMPLE_COUNT_NAME

Source code in tensorflow_model_analysis/metrics/weighted_example_count.py

def __init__(self, name: str = WEIGHTED_EXAMPLE_COUNT_NAME):
  """Initializes weighted example count.

  Args:
    name: Metric name.
  """

  super().__init__(name=name)

Attributes

compute_confidence_interval property

compute_confidence_interval: bool

Always disable confidence intervals for ExampleCount.

Confidence intervals capture uncertainty in a metric if it were computed on more examples. For ExampleCount, this sort of uncertainty is not meaningful, so confidence intervals are disabled.

RETURNS	DESCRIPTION
bool	Whether to compute confidence intervals.

create_computations_fn instance-attribute

create_computations_fn = create_computations_fn

kwargs instance-attribute

kwargs = kwargs

name instance-attribute

name = name

Functions

computations

computations(
    eval_config: Optional[EvalConfig] = None,
    schema: Optional[Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations

Creates computations associated with metric.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def computations(
    self,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    schema: Optional[schema_pb2.Schema] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    sub_keys: Optional[List[Optional[SubKey]]] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    query_key: Optional[str] = None,
) -> MetricComputations:
  """Creates computations associated with metric."""
  updated_kwargs = validate_and_update_create_computations_fn_kwargs(
      self._args,
      self.kwargs.copy(),
      eval_config,
      schema,
      model_names,
      output_names,
      sub_keys,
      aggregation_type,
      class_weights,
      example_weighted,
      query_key,
  )
  return self.create_computations_fn(**updated_kwargs)

from_config classmethod

from_config(config: Dict[str, Any]) -> Metric

Source code in tensorflow_model_analysis/metrics/metric_types.py

@classmethod
def from_config(cls, config: Dict[str, Any]) -> 'Metric':
  # `fn` key is unnecessary for wrapper due to
  # `create_computation_fn` key serialization.
  config.pop('fn', None)
  return cls(**config)

get_config

get_config() -> Dict[str, Any]

Returns serializable config.

Source code in tensorflow_model_analysis/metrics/metric_types.py

def get_config(self) -> Dict[str, Any]:
  """Returns serializable config."""
  return self.kwargs

Functions

CombinedFeaturePreprocessor

CombinedFeaturePreprocessor(
    feature_keys: Iterable[str],
    include_default_inputs: bool = True,
    model_names: Optional[Iterable[str]] = None,
    output_names: Optional[Iterable[str]] = None,
) -> StandardMetricInputsPreprocessor

Returns preprocessor for incl combined features in StandardMetricInputs.

PARAMETER	DESCRIPTION
feature_keys	List of feature keys. An empty list means all. TYPE: Iterable[str]
include_default_inputs	True to include default inputs (labels, predictions, example weights) in addition to the transformed features. TYPE: bool DEFAULT: True
model_names	Optional model names (required if transformed_features used with multi-model evaluations). TYPE: Optional[Iterable[str]] DEFAULT: None
output_names	Optional output names. Only used if include_default_inputs is True. If unset all outputs will be included with the default inputs. TYPE: Optional[Iterable[str]] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/metric_types.py

def CombinedFeaturePreprocessor(  # pylint: disable=invalid-name
    feature_keys: Iterable[str],
    include_default_inputs: bool = True,
    model_names: Optional[Iterable[str]] = None,
    output_names: Optional[Iterable[str]] = None,
) -> StandardMetricInputsPreprocessor:
  """Returns preprocessor for incl combined features in StandardMetricInputs.

  Args:
    feature_keys: List of feature keys. An empty list means all.
    include_default_inputs: True to include default inputs (labels, predictions,
      example weights) in addition to the transformed features.
    model_names: Optional model names (required if transformed_features used
      with multi-model evaluations).
    output_names: Optional output names. Only used if include_default_inputs is
      True. If unset all outputs will be included with the default inputs.
  """
  if feature_keys:
    include_features = {k: {} for k in feature_keys}
  else:
    include_features = {}
  if model_names:
    include_features = {name: include_features for name in model_names}
  return StandardMetricInputsPreprocessor(
      include_filter={
          constants.TRANSFORMED_FEATURES_KEY: include_features,
          constants.FEATURES_KEY: include_features,
      },
      include_default_inputs=include_default_inputs,
      model_names=model_names,
      output_names=output_names,
      name=_DEFAULT_COMBINED_FEATURE_PREPROCESSOR_NAME,
  )

FeaturePreprocessor

FeaturePreprocessor(
    feature_keys: Iterable[str],
    include_default_inputs: bool = True,
    model_names: Optional[Iterable[str]] = None,
    output_names: Optional[Iterable[str]] = None,
) -> StandardMetricInputsPreprocessor

Returns preprocessor for including features in StandardMetricInputs.

PARAMETER	DESCRIPTION
feature_keys	List of feature keys. An empty list means all. TYPE: Iterable[str]
include_default_inputs	True to include default inputs (labels, predictions, example weights) in addition to the features. TYPE: bool DEFAULT: True
model_names	Optional model names. Only used if include_default_inputs is True. If unset all models will be included with the default inputs. TYPE: Optional[Iterable[str]] DEFAULT: None
output_names	Optional output names. Only used if include_default_inputs is True. If unset all outputs will be included with the default inputs. TYPE: Optional[Iterable[str]] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/metric_types.py

def FeaturePreprocessor(  # pylint: disable=invalid-name
    feature_keys: Iterable[str],
    include_default_inputs: bool = True,
    model_names: Optional[Iterable[str]] = None,
    output_names: Optional[Iterable[str]] = None,
) -> StandardMetricInputsPreprocessor:
  """Returns preprocessor for including features in StandardMetricInputs.

  Args:
    feature_keys: List of feature keys. An empty list means all.
    include_default_inputs: True to include default inputs (labels, predictions,
      example weights) in addition to the features.
    model_names: Optional model names. Only used if include_default_inputs is
      True. If unset all models will be included with the default inputs.
    output_names: Optional output names. Only used if include_default_inputs is
      True. If unset all outputs will be included with the default inputs.
  """
  if feature_keys:
    include_features = {k: {} for k in feature_keys}
  else:
    include_features = {}
  return StandardMetricInputsPreprocessor(
      include_filter={constants.FEATURES_KEY: include_features},
      include_default_inputs=include_default_inputs,
      model_names=model_names,
      output_names=output_names,
      name=_DEFAULT_FEATURE_PREPROCESSOR_NAME,
  )

default_binary_classification_specs

default_binary_classification_specs(
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    binarize: Optional[BinarizationOptions] = None,
    aggregate: Optional[AggregationOptions] = None,
    include_loss: bool = True,
) -> List[MetricsSpec]

Returns default metric specs for binary classification problems.

PARAMETER	DESCRIPTION
model_names	Optional model names (if multi-model evaluation). TYPE: Optional[List[str]] DEFAULT: None
output_names	Optional list of output names (if multi-output model). TYPE: Optional[List[str]] DEFAULT: None
output_weights	Optional output weights for creating overall metric aggregated across outputs (if multi-output model). If a weight is not provided for an output, it's weight defaults to 0.0 (i.e. output ignored). TYPE: Optional[Dict[str, float]] DEFAULT: None
binarize	Optional settings for binarizing multi-class/multi-label metrics. TYPE: Optional[BinarizationOptions] DEFAULT: None
aggregate	Optional settings for aggregating multi-class/multi-label metrics. TYPE: Optional[AggregationOptions] DEFAULT: None
include_loss	True to include loss. TYPE: bool DEFAULT: True

Source code in tensorflow_model_analysis/metrics/metric_specs.py

def default_binary_classification_specs(
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    binarize: Optional[config_pb2.BinarizationOptions] = None,
    aggregate: Optional[config_pb2.AggregationOptions] = None,
    include_loss: bool = True) -> List[config_pb2.MetricsSpec]:
  """Returns default metric specs for binary classification problems.

  Args:
    model_names: Optional model names (if multi-model evaluation).
    output_names: Optional list of output names (if multi-output model).
    output_weights: Optional output weights for creating overall metric
      aggregated across outputs (if multi-output model). If a weight is not
      provided for an output, it's weight defaults to 0.0 (i.e. output ignored).
    binarize: Optional settings for binarizing multi-class/multi-label metrics.
    aggregate: Optional settings for aggregating multi-class/multi-label
      metrics.
    include_loss: True to include loss.
  """

  metrics = [
      confusion_matrix_metrics.BinaryAccuracy(name='binary_accuracy'),
      confusion_matrix_metrics.AUC(
          name='auc',
          num_thresholds=binary_confusion_matrices.DEFAULT_NUM_THRESHOLDS),
      confusion_matrix_metrics.AUC(
          name='auc_precison_recall',  # Matches default name used by estimator.
          curve='PR',
          num_thresholds=binary_confusion_matrices.DEFAULT_NUM_THRESHOLDS),
      confusion_matrix_metrics.Precision(name='precision'),
      confusion_matrix_metrics.Recall(name='recall'),
      calibration.MeanLabel(name='mean_label'),
      calibration.MeanPrediction(name='mean_prediction'),
      calibration.Calibration(name='calibration'),
      confusion_matrix_plot.ConfusionMatrixPlot(name='confusion_matrix_plot'),
      calibration_plot.CalibrationPlot(name='calibration_plot')
  ]
  if include_loss:
    metrics.append(tf_keras.metrics.BinaryCrossentropy(name='loss'))

  return specs_from_metrics(
      metrics,
      model_names=model_names,
      output_names=output_names,
      output_weights=output_weights,
      binarize=binarize,
      aggregate=aggregate)

default_multi_class_classification_specs

default_multi_class_classification_specs(
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    binarize: Optional[BinarizationOptions] = None,
    aggregate: Optional[AggregationOptions] = None,
    sparse: bool = True,
) -> List[MetricsSpec]

Returns default metric specs for multi-class classification problems.

PARAMETER	DESCRIPTION
model_names	Optional model names if multi-model evaluation. TYPE: Optional[List[str]] DEFAULT: None
output_names	Optional list of output names (if multi-output model). TYPE: Optional[List[str]] DEFAULT: None
output_weights	Optional output weights for creating overall metric aggregated across outputs (if multi-output model). If a weight is not provided for an output, it's weight defaults to 0.0 (i.e. output ignored). TYPE: Optional[Dict[str, float]] DEFAULT: None
binarize	Optional settings for binarizing multi-class/multi-label metrics. TYPE: Optional[BinarizationOptions] DEFAULT: None
aggregate	Optional settings for aggregating multi-class/multi-label metrics. TYPE: Optional[AggregationOptions] DEFAULT: None
sparse	True if the labels are sparse. TYPE: bool DEFAULT: True

Source code in tensorflow_model_analysis/metrics/metric_specs.py

def default_multi_class_classification_specs(
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    binarize: Optional[config_pb2.BinarizationOptions] = None,
    aggregate: Optional[config_pb2.AggregationOptions] = None,
    sparse: bool = True) -> List[config_pb2.MetricsSpec]:
  """Returns default metric specs for multi-class classification problems.

  Args:
    model_names: Optional model names if multi-model evaluation.
    output_names: Optional list of output names (if multi-output model).
    output_weights: Optional output weights for creating overall metric
      aggregated across outputs (if multi-output model). If a weight is not
      provided for an output, it's weight defaults to 0.0 (i.e. output ignored).
    binarize: Optional settings for binarizing multi-class/multi-label metrics.
    aggregate: Optional settings for aggregating multi-class/multi-label
      metrics.
    sparse: True if the labels are sparse.
  """

  if sparse:
    metrics = [
        tf_keras.metrics.SparseCategoricalCrossentropy(name='loss'),
        tf_keras.metrics.SparseCategoricalAccuracy(name='accuracy'),
    ]
  else:
    metrics = [
        tf_keras.metrics.CategoricalCrossentropy(name='loss'),
        tf_keras.metrics.CategoricalAccuracy(name='accuracy'),
    ]
  metrics.append(
      multi_class_confusion_matrix_plot.MultiClassConfusionMatrixPlot())
  if binarize is not None:
    for top_k in binarize.top_k_list.values:
      metrics.extend([
          confusion_matrix_metrics.Precision(name='precision', top_k=top_k),
          confusion_matrix_metrics.Recall(name='recall', top_k=top_k)
      ])
    binarize_without_top_k = config_pb2.BinarizationOptions()
    binarize_without_top_k.CopyFrom(binarize)
    binarize_without_top_k.ClearField('top_k_list')
    binarize = binarize_without_top_k
  multi_class_metrics = specs_from_metrics(
      metrics,
      model_names=model_names,
      output_names=output_names,
      output_weights=output_weights)
  if aggregate is None:
    aggregate = config_pb2.AggregationOptions(micro_average=True)
  multi_class_metrics.extend(
      default_binary_classification_specs(
          model_names=model_names,
          output_names=output_names,
          output_weights=output_weights,
          binarize=binarize,
          aggregate=aggregate))
  return multi_class_metrics

default_regression_specs

default_regression_specs(
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    loss_functions: Optional[
        List[Union[Metric, Loss]]
    ] = None,
    min_value: Optional[float] = None,
    max_value: Optional[float] = None,
) -> List[MetricsSpec]

Returns default metric specs for for regression problems.

PARAMETER	DESCRIPTION
model_names	Optional model names (if multi-model evaluation). TYPE: Optional[List[str]] DEFAULT: None
output_names	Optional list of output names (if multi-output model). TYPE: Optional[List[str]] DEFAULT: None
output_weights	Optional output weights for creating overall metric aggregated across outputs (if multi-output model). If a weight is not provided for an output, it's weight defaults to 0.0 (i.e. output ignored). TYPE: Optional[Dict[str, float]] DEFAULT: None
loss_functions	Loss functions to use (if None MSE is used). TYPE: Optional[List[Union[Metric, Loss]]] DEFAULT: None
min_value	Min value for calibration plot (if None no plot will be created). TYPE: Optional[float] DEFAULT: None
max_value	Max value for calibration plot (if None no plot will be created). TYPE: Optional[float] DEFAULT: None

Source code in tensorflow_model_analysis/metrics/metric_specs.py

def default_regression_specs(
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    loss_functions: Optional[
        List[Union[tf_keras.metrics.Metric, tf_keras.losses.Loss]]
    ] = None,
    min_value: Optional[float] = None,
    max_value: Optional[float] = None,
) -> List[config_pb2.MetricsSpec]:
  """Returns default metric specs for for regression problems.

  Args:
    model_names: Optional model names (if multi-model evaluation).
    output_names: Optional list of output names (if multi-output model).
    output_weights: Optional output weights for creating overall metric
      aggregated across outputs (if multi-output model). If a weight is not
      provided for an output, it's weight defaults to 0.0 (i.e. output ignored).
    loss_functions: Loss functions to use (if None MSE is used).
    min_value: Min value for calibration plot (if None no plot will be created).
    max_value: Max value for calibration plot (if None no plot will be created).
  """

  if loss_functions is None:
    loss_functions = [tf_keras.metrics.MeanSquaredError(name='mse')]

  metrics = [
      tf_keras.metrics.Accuracy(name='accuracy'),
      calibration.MeanLabel(name='mean_label'),
      calibration.MeanPrediction(name='mean_prediction'),
      calibration.Calibration(name='calibration'),
  ]
  for fn in loss_functions:
    metrics.append(fn)
  if min_value is not None and max_value is not None:
    metrics.append(
        calibration_plot.CalibrationPlot(
            name='calibration_plot', left=min_value, right=max_value))

  return specs_from_metrics(
      metrics,
      model_names=model_names,
      output_names=output_names,
      output_weights=output_weights)

has_attributions_metrics

has_attributions_metrics(
    metrics_specs: Iterable[MetricsSpec],
) -> bool

Returns true if any of the metrics_specs have attributions metrics.

Source code in tensorflow_model_analysis/metrics/attributions.py

def has_attributions_metrics(
    metrics_specs: Iterable[config_pb2.MetricsSpec]) -> bool:
  """Returns true if any of the metrics_specs have attributions metrics."""
  tfma_metric_classes = metric_types.registered_metrics()
  for metrics_spec in metrics_specs:
    for metric_config in metrics_spec.metrics:
      instance = metric_specs.metric_instance(metric_config,
                                              tfma_metric_classes)
      if isinstance(instance, AttributionsMetric):
        return True
  return False

merge_per_key_computations

merge_per_key_computations(
    create_computations_fn: Callable[
        ..., MetricComputations
    ],
) -> Callable[..., MetricComputations]

Wraps create_computations_fn to be called separately for each key.

Source code in tensorflow_model_analysis/metrics/metric_util.py

def merge_per_key_computations(
    create_computations_fn: Callable[..., metric_types.MetricComputations],
) -> Callable[..., metric_types.MetricComputations]:
  """Wraps create_computations_fn to be called separately for each key."""

  def merge_computations_fn(
      eval_config: Optional[config_pb2.EvalConfig] = None,
      schema: Optional[schema_pb2.Schema] = None,
      model_names: Optional[List[str]] = None,
      output_names: Optional[List[str]] = None,
      sub_keys: Optional[List[Optional[metric_types.SubKey]]] = None,
      aggregation_type: Optional[metric_types.AggregationType] = None,
      class_weights: Optional[Dict[int, float]] = None,
      example_weighted: bool = False,
      query_key: Optional[str] = None,
      **kwargs) -> metric_types.MetricComputations:
    """Merge computations function."""
    if model_names is None:
      model_names = ['']
    if output_names is None:
      output_names = ['']
    if sub_keys is None:
      sub_keys = [None]
    computations = []
    for model_name in model_names:
      for output_name in output_names:
        for sub_key in sub_keys:
          if hasattr(inspect, 'getfullargspec'):
            args = inspect.getfullargspec(create_computations_fn).args
          else:
            args = inspect.getargspec(create_computations_fn).args  # pylint: disable=deprecated-method
          updated_kwargs = metric_types.validate_and_update_create_computations_fn_kwargs(
              args, kwargs.copy(), eval_config, schema, model_names,
              output_names, sub_keys, aggregation_type, class_weights,
              example_weighted, query_key)
          if 'model_name' in args:
            updated_kwargs['model_name'] = model_name
          if 'output_name' in args:
            updated_kwargs['output_name'] = output_name
          if 'sub_key' in args:
            updated_kwargs['sub_key'] = sub_key
          computations.extend(create_computations_fn(**updated_kwargs))
    return computations

  return merge_computations_fn

metric_thresholds_from_metrics_specs

metric_thresholds_from_metrics_specs(
    metrics_specs: Iterable[MetricsSpec],
    eval_config: Optional[EvalConfig] = None,
) -> Dict[MetricKey, Iterable[_SliceAndThreshold]]

Returns thresholds associated with given metrics specs.

Source code in tensorflow_model_analysis/metrics/metric_specs.py

def metric_thresholds_from_metrics_specs(
    metrics_specs: Iterable[config_pb2.MetricsSpec],
    eval_config: Optional[config_pb2.EvalConfig] = None
) -> Dict[metric_types.MetricKey, Iterable[_SliceAndThreshold]]:
  """Returns thresholds associated with given metrics specs."""
  if eval_config is None:
    eval_config = config_pb2.EvalConfig()
  result = collections.defaultdict(list)
  existing = collections.defaultdict(dict)

  def add_if_not_exists(
      key: metric_types.MetricKey,
      slice_spec: Optional[Union[config_pb2.SlicingSpec,
                                 config_pb2.CrossSlicingSpec]],
      threshold: Union[config_pb2.GenericChangeThreshold,
                       config_pb2.GenericValueThreshold]):
    """Adds value to results if it doesn't already exist."""
    hashable_slice_spec = None
    if slice_spec:
      hashable_slice_spec = slicer.deserialize_slice_spec(slice_spec)
    # Note that hashing by SerializeToString() is only safe if used within the
    # same process.
    threshold_hash = threshold.SerializeToString()
    if (not (key in existing and hashable_slice_spec in existing[key] and
             threshold_hash in existing[key][hashable_slice_spec])):
      if hashable_slice_spec not in existing[key]:
        existing[key][hashable_slice_spec] = {}
      existing[key][hashable_slice_spec][threshold_hash] = True
      result[key].append((slice_spec, threshold))

  def add_threshold(key: metric_types.MetricKey,
                    slice_spec: Union[Optional[config_pb2.SlicingSpec],
                                      Optional[config_pb2.CrossSlicingSpec]],
                    threshold: config_pb2.MetricThreshold):
    """Adds thresholds to results."""
    if threshold.HasField('value_threshold'):
      add_if_not_exists(key, slice_spec, threshold.value_threshold)
    if threshold.HasField('change_threshold'):
      key = key.make_diff_key()
      add_if_not_exists(key, slice_spec, threshold.change_threshold)

  for spec in metrics_specs:
    for aggregation_type, sub_keys in _create_sub_keys(spec).items():
      # Add thresholds for metrics computed in-graph.
      for metric_name, threshold in spec.thresholds.items():
        for key in _keys_for_metric(metric_name, spec, aggregation_type,
                                    sub_keys, [None]):
          add_threshold(key, None, threshold)
      for metric_name, per_slice_thresholds in spec.per_slice_thresholds.items(
      ):
        for key in _keys_for_metric(metric_name, spec, aggregation_type,
                                    sub_keys, [None]):
          for per_slice_threshold in per_slice_thresholds.thresholds:
            for slice_spec in per_slice_threshold.slicing_specs:
              add_threshold(key, slice_spec, per_slice_threshold.threshold)
      for metric_name, cross_slice_thresholds in (
          spec.cross_slice_thresholds.items()):
        for key in _keys_for_metric(metric_name, spec, aggregation_type,
                                    sub_keys, [None]):
          for cross_slice_threshold in cross_slice_thresholds.thresholds:
            for cross_slice_spec in cross_slice_threshold.cross_slicing_specs:
              add_threshold(key, cross_slice_spec,
                            cross_slice_threshold.threshold)

  # Add thresholds for post export metrics defined in MetricConfigs.
  for key, metric_config, _ in keys_and_metrics_from_specs(
      eval_config, metrics_specs):
    if metric_config.HasField('threshold'):
      add_threshold(key, None, metric_config.threshold)
    for per_slice_threshold in metric_config.per_slice_thresholds:
      for slice_spec in per_slice_threshold.slicing_specs:
        add_threshold(key, slice_spec, per_slice_threshold.threshold)
    for cross_slice_threshold in metric_config.cross_slice_thresholds:
      for cross_slice_spec in cross_slice_threshold.cross_slicing_specs:
        add_threshold(key, cross_slice_spec, cross_slice_threshold.threshold)

  return result

specs_from_metrics

specs_from_metrics(
    metrics: Optional[_MetricsOrLosses] = None,
    unweighted_metrics: Optional[_MetricsOrLosses] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    binarize: Optional[BinarizationOptions] = None,
    aggregate: Optional[AggregationOptions] = None,
    query_key: Optional[str] = None,
    include_example_count: Optional[bool] = None,
    include_weighted_example_count: Optional[bool] = None,
) -> List[MetricsSpec]

Returns specs for tf_keras.metrics/losses or tfma.metrics classes.

Examples:

metrics_specs = specs_from_metrics( [ tf_keras.metrics.BinaryAccuracy(), tfma.metrics.AUC(), tfma.metrics.MeanLabel(), tfma.metrics.MeanPrediction() ... ], unweighted=[ tfma.metrics.Precision(), tfma.metrics.Recall() ])

metrics_specs = specs_from_metrics({ 'output1': [ tf_keras.metrics.BinaryAccuracy(), tfma.metrics.AUC(), tfma.metrics.MeanLabel(), tfma.metrics.MeanPrediction() ... ], 'output2': [ tfma.metrics.Precision(), tfma.metrics.Recall(), ] })

PARAMETER	DESCRIPTION
metrics	List of tfma.metrics.Metric, tf_keras.metrics.Metric, or tf_keras.losses.Loss. For multi-output models a dict of dicts may be passed where the first dict is indexed by the output_name. Whether these metrics are weighted or not will be determined based on whether the ModelSpec associated with the metrics contains example weight key settings or not. TYPE: Optional[_MetricsOrLosses] DEFAULT: None
unweighted_metrics	Same as metrics only these metrics will not be weighted by example_weight regardless of the example weight key settings. TYPE: Optional[_MetricsOrLosses] DEFAULT: None
model_names	Optional model names (if multi-model evaluation). TYPE: Optional[List[str]] DEFAULT: None
output_names	Optional output names (if multi-output models). If the metrics are a dict this should not be set. TYPE: Optional[List[str]] DEFAULT: None
output_weights	Optional output weights for creating overall metric aggregated across outputs (if multi-output model). If a weight is not provided for an output, it's weight defaults to 0.0 (i.e. output ignored). TYPE: Optional[Dict[str, float]] DEFAULT: None
binarize	Optional settings for binarizing multi-class/multi-label metrics. TYPE: Optional[BinarizationOptions] DEFAULT: None
aggregate	Optional settings for aggregating multi-class/multi-label metrics. TYPE: Optional[AggregationOptions] DEFAULT: None
query_key	Optional query key for query/ranking based metrics. TYPE: Optional[str] DEFAULT: None
include_example_count	True to add example_count metric. Default is True. TYPE: Optional[bool] DEFAULT: None
include_weighted_example_count	True to add weighted example_count metric. Default is True. A weighted example count will be added per output for multi-output models. TYPE: Optional[bool] DEFAULT: None

RETURNS	DESCRIPTION
List[MetricsSpec]	MetricsSpecs based on options provided. A separate spec is returned for
List[MetricsSpec]	weighted vs unweighted metrics. A separate spec is also returned for each
List[MetricsSpec]	output if a dict of metrics per output is passed.

Source code in tensorflow_model_analysis/metrics/metric_specs.py

def specs_from_metrics(
    metrics: Optional[_MetricsOrLosses] = None,
    unweighted_metrics: Optional[_MetricsOrLosses] = None,
    model_names: Optional[List[str]] = None,
    output_names: Optional[List[str]] = None,
    output_weights: Optional[Dict[str, float]] = None,
    binarize: Optional[config_pb2.BinarizationOptions] = None,
    aggregate: Optional[config_pb2.AggregationOptions] = None,
    query_key: Optional[str] = None,
    include_example_count: Optional[bool] = None,
    include_weighted_example_count: Optional[bool] = None
) -> List[config_pb2.MetricsSpec]:
  """Returns specs for tf_keras.metrics/losses or tfma.metrics classes.

  Examples:

    metrics_specs = specs_from_metrics(
      [
          tf_keras.metrics.BinaryAccuracy(),
          tfma.metrics.AUC(),
          tfma.metrics.MeanLabel(),
          tfma.metrics.MeanPrediction()
          ...
      ],
      unweighted=[
          tfma.metrics.Precision(),
          tfma.metrics.Recall()
      ])

    metrics_specs = specs_from_metrics({
      'output1': [
          tf_keras.metrics.BinaryAccuracy(),
          tfma.metrics.AUC(),
          tfma.metrics.MeanLabel(),
          tfma.metrics.MeanPrediction()
          ...
      ],
      'output2': [
          tfma.metrics.Precision(),
          tfma.metrics.Recall(),
      ]
    })

  Args:
    metrics: List of tfma.metrics.Metric, tf_keras.metrics.Metric, or
      tf_keras.losses.Loss. For multi-output models a dict of dicts may be
      passed where the first dict is indexed by the output_name. Whether these
      metrics are weighted or not will be determined based on whether the
      ModelSpec associated with the metrics contains example weight key settings
      or not.
    unweighted_metrics: Same as metrics only these metrics will not be weighted
      by example_weight regardless of the example weight key settings.
    model_names: Optional model names (if multi-model evaluation).
    output_names: Optional output names (if multi-output models). If the metrics
      are a dict this should not be set.
    output_weights: Optional output weights for creating overall metric
      aggregated across outputs (if multi-output model). If a weight is not
      provided for an output, it's weight defaults to 0.0 (i.e. output ignored).
    binarize: Optional settings for binarizing multi-class/multi-label metrics.
    aggregate: Optional settings for aggregating multi-class/multi-label
      metrics.
    query_key: Optional query key for query/ranking based metrics.
    include_example_count: True to add example_count metric. Default is True.
    include_weighted_example_count: True to add weighted example_count metric.
      Default is True. A weighted example count will be added per output for
      multi-output models.

  Returns:
    MetricsSpecs based on options provided. A separate spec is returned for
    weighted vs unweighted metrics. A separate spec is also returned for each
    output if a dict of metrics per output is passed.
  """
  if isinstance(metrics, dict) and output_names:
    raise ValueError('metrics cannot be a dict when output_names is used: '
                     'metrics={}, output_names={}'.format(
                         metrics, output_names))
  if (metrics and unweighted_metrics and
      isinstance(metrics, dict) != isinstance(unweighted_metrics, dict)):
    raise ValueError(
        'metrics and unweighted_metrics must both be either dicts or lists: '
        f'metrics={metrics}, unweighted_metrics={unweighted_metrics}')

  if isinstance(metrics, dict) or isinstance(unweighted_metrics, dict):
    metrics_dict = metrics if isinstance(metrics, dict) else {}
    unweighted_metrics_dict = (
        unweighted_metrics if isinstance(unweighted_metrics, dict) else {})
    specs = []
    output_names = set(metrics_dict) | set(unweighted_metrics_dict)
    for output_name in sorted(output_names):
      specs.extend(
          specs_from_metrics(
              metrics_dict.get(output_name),
              unweighted_metrics=unweighted_metrics_dict.get(output_name),
              model_names=model_names,
              output_names=[output_name],
              binarize=binarize,
              aggregate=aggregate,
              include_example_count=include_example_count,
              include_weighted_example_count=include_weighted_example_count))
      include_example_count = False
    return specs

  if include_example_count is None:
    include_example_count = True
  if include_weighted_example_count is None:
    include_weighted_example_count = True

  # Add the computations for the example counts and weights since they are
  # independent of the model and class ID.
  specs = example_count_specs(
      model_names=model_names,
      output_names=output_names,
      output_weights=output_weights,
      include_example_count=include_example_count,
      include_weighted_example_count=include_weighted_example_count)

  if metrics:
    specs.append(
        config_pb2.MetricsSpec(
            metrics=[config_from_metric(metric) for metric in metrics],
            model_names=model_names,
            output_names=output_names,
            output_weights=output_weights,
            binarize=binarize,
            aggregate=aggregate,
            example_weights=None,
            query_key=query_key))
  if unweighted_metrics:
    specs.append(
        config_pb2.MetricsSpec(
            metrics=[
                config_from_metric(metric) for metric in unweighted_metrics
            ],
            model_names=model_names,
            output_names=output_names,
            output_weights=output_weights,
            binarize=binarize,
            aggregate=aggregate,
            example_weights=config_pb2.ExampleWeightOptions(unweighted=True),
            query_key=query_key))

  return specs

to_label_prediction_example_weight

to_label_prediction_example_weight(
    inputs: StandardMetricInputs,
    eval_config: Optional[EvalConfig] = None,
    model_name: str = "",
    output_name: str = "",
    sub_key: Optional[SubKey] = None,
    aggregation_type: Optional[AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    fractional_labels: bool = False,
    flatten: bool = True,
    squeeze: bool = True,
    allow_none: bool = False,
    require_single_example_weight: bool = False,
) -> Iterator[Tuple[ndarray, ndarray, ndarray]]

Yields label, prediction, and example weights for use in calculations.

Where applicable this function will perform model and output name lookups as well as any required class ID, top K, etc conversions. It will also apply prediction keys and label vocabularies given the necessary information is provided as part of the EvalConfig (or standard estimator based naming is used). The sparseness of labels will be inferred from the shapes of the labels and predictions (i.e. if the shapes are different then the labels will be assumed to be sparse).

If successful, the final output of calling this function will be a tuple of numpy arrays representing the label, prediction, and example weight respectively. Labels and predictions will be returned in the same shape provided (default behavior) unless (1) flatten is True in which case a series of values (one per class ID) will be returned with last dimension of size 1 or (2) a sub_key is used in which case the last dimension may be re-shaped to match the new number of outputs (1 for class_id or k, top_k for top k with aggregation).

Note that for top_k without aggregation, the non-top_k prediction values will be set to float('-inf'), but for top_k with aggregation the values will be truncated to only return the top k values.

Examples:

# default behavior # # Binary classification Input : labels=[1] predictions=[0.6] Output : (np.array([1]), np.array([0.6]), np.array([1.0])) # Multi-class classification w/ sparse labels Input : labels=[2] predictions=[0.3, 0.6, 0.1] Output: (np.array([2]), np.array([0.3, 0.6, 0.1]), np.array([1.0])) # Multi-class / multi-label classification w/ dense labels Input : labels=[0, 1, 1] predictions=[0.3, 0.6, 0.1] Output : (np.array([0, 1, 1]), np.array([0.3, 0.6, 0.1]), np.array([1.0]))

# flatten=True # # Multi-class classification w/ sparse labels Input : labels=[2], predictions=[0.3, 0.6, 0.1] Output : (np.array([0]), np.array([0.3]), np.array([1.0])), (np.array([0]), np.array([0.6]), np.array([1.0])), (np.array([1]), np.array([0.1]), np.array([1.0])) # Multi-class/multi-label classification w/ dense labels Input : labels=[0, 0, 1], predictions=[0.3, 0.6, 0.1] Output : (np.array([0]), np.array([0.3]), np.array([1.0])), (np.array([0]), np.array([0.6]), np.array([1.0])), (np.array([1]), np.array([0.1]), np.array([1.0]))

# sub_key.class_id=[2] # # Multi-class classification w/ sparse labels Input : labels=[2] predictions=[0.3, 0.6, 0.1] Output : (np.array([1]), np.array([0.1]), np.array([1.0])) # Multi-class classification w/ dense labels Input : labels=[0, 0, 1] predictions=[0.3, 0.6, 0.1] Output : (np.array([1]), np.array([0.1]), np.array([1.0]))

# sub_key.top_k=2 and aggregation_type is None (i.e. binarization of top 2). # # Multi-class classification w/ sparse labels Input : labels=[2] predictions=[0.3, 0.6, 0.1] Output : (np.array([0, 0, 1]), np.array([0.3, 0.6, -inf]), np.array([1.0])) # Multi-class classification w/ dense labels Input : labels=[0, 0, 1] predictions=[0.3, 0.1, 0.6] Output : (np.array([0, 0, 1]), np.array([0.3, -inf, 0.6]), np.array([1.0]))

# sub_key.top_k=2 and aggregation_type is not None (i.e. aggregate top 2). # # Multi-class classification w/ sparse labels Input : labels=[2] predictions=[0.3, 0.6, 0.1] Output : (np.array([0, 1]), np.array([0.3, 0.6]), np.array([1.0])) # Multi-class classification w/ dense labels Input : labels=[0, 0, 1] predictions=[0.3, 0.1, 0.6] Output : (np.array([0, 0]), np.array([0.3, 0.6]), np.array([1.0]))

# sub_key.k=2 (i.e. binarization by choosing 2nd largest predicted value). # # Multi-class classification w/ sparse labels Input : labels=[0] predictions=[0.3, 0.6, 0.1] Output : (np.array([1]), np.array([0.3]), np.array([1.0])) # Multi-class classification w/ dense labels Input : labels=[0] predictions=[0.3] Output : (np.array([0]), np.array([0.3]), np.array([1.0]))

PARAMETER	DESCRIPTION
inputs	Standard metric inputs. TYPE: StandardMetricInputs
eval_config	Eval config TYPE: Optional[EvalConfig] DEFAULT: None
model_name	Optional model name (if multi-model evaluation). TYPE: str DEFAULT: ''
output_name	Optional output name (if multi-output model type). TYPE: str DEFAULT: ''
sub_key	Optional sub key. TYPE: Optional[SubKey] DEFAULT: None
aggregation_type	Optional aggregation type. TYPE: Optional[AggregationType] DEFAULT: None
class_weights	Optional class weights to apply to multi-class / multi-label labels and predictions. If used, flatten must also be True. TYPE: Optional[Dict[int, float]] DEFAULT: None
example_weighted	True if example weights should be applied. TYPE: bool DEFAULT: False
fractional_labels	If true, each incoming tuple of (label, prediction, and example weight) will be split into two tuples as follows (where l, p, w represent the resulting label, prediction, and example weight values): (1) l = 0.0, p = prediction, and w = example_weight * (1.0 - label) (2) l = 1.0, p = prediction, and w = example_weight * label If enabled, an exception will be raised if labels are not within [0, 1]. The implementation is such that tuples associated with a weight of zero are not yielded. This means it is safe to enable fractional_labels even when the labels only take on the values of 0.0 or 1.0. TYPE: bool DEFAULT: False
flatten	True to flatten the final label and prediction outputs so that the yielded values are always arrays of size 1. For example, multi-class / multi-label outputs would be converted into label and prediction pairs that could then be processed by a binary classification metric in order to compute a micro average over all classes. If the example weight is not a scalar, then they will be flattened as well, otherwise the same example weight value will be output for each pair of labels and predictions. TYPE: bool DEFAULT: True
squeeze	True to squeeze any outputs that have rank > 1. This transforms outputs such as np.array([[1]]) to np.array([1]). TYPE: bool DEFAULT: True
allow_none	True to allow labels or predictions with None values to be returned. When used, the values will be returned as empty np.ndarrays. The example weight will always be non-empty. TYPE: bool DEFAULT: False
require_single_example_weight	True to require that the example_weight be a single value. TYPE: bool DEFAULT: False

YIELDS	DESCRIPTION
Tuple[ndarray, ndarray, ndarray]	Tuple of (label, prediction, example_weight).

Source code in tensorflow_model_analysis/metrics/metric_util.py

def to_label_prediction_example_weight(
    inputs: metric_types.StandardMetricInputs,
    eval_config: Optional[config_pb2.EvalConfig] = None,
    model_name: str = '',
    output_name: str = '',
    sub_key: Optional[metric_types.SubKey] = None,
    aggregation_type: Optional[metric_types.AggregationType] = None,
    class_weights: Optional[Dict[int, float]] = None,
    example_weighted: bool = False,
    fractional_labels: bool = False,
    flatten: bool = True,
    squeeze: bool = True,
    allow_none: bool = False,
    require_single_example_weight: bool = False
) -> Iterator[Tuple[np.ndarray, np.ndarray, np.ndarray]]:
  """Yields label, prediction, and example weights for use in calculations.

  Where applicable this function will perform model and output name lookups as
  well as any required class ID, top K, etc conversions. It will also apply
  prediction keys and label vocabularies given the necessary information is
  provided as part of the EvalConfig (or standard estimator based naming is
  used). The sparseness of labels will be inferred from the shapes of the labels
  and predictions (i.e. if the shapes are different then the labels will be
  assumed to be sparse).

  If successful, the final output of calling this function will be a tuple of
  numpy arrays representing the label, prediction, and example weight
  respectively. Labels and predictions will be returned in the same shape
  provided (default behavior) unless (1) flatten is True in which case a series
  of values (one per class ID) will be returned with last dimension of size 1 or
  (2) a sub_key is used in which case the last dimension may be re-shaped to
  match the new number of outputs (1 for class_id or k, top_k for top k with
  aggregation).

  Note that for top_k without aggregation, the non-top_k prediction values will
  be set to float('-inf'), but for top_k with aggregation the values will be
  truncated to only return the top k values.

  Examples:

    # default behavior
    #
    # Binary classification
    Input  : labels=[1] predictions=[0.6]
    Output : (np.array([1]), np.array([0.6]), np.array([1.0]))
    # Multi-class classification w/ sparse labels
    Input : labels=[2] predictions=[0.3, 0.6, 0.1]
    Output: (np.array([2]), np.array([0.3, 0.6, 0.1]), np.array([1.0]))
    # Multi-class / multi-label classification w/ dense labels
    Input  : labels=[0, 1, 1] predictions=[0.3, 0.6, 0.1]
    Output : (np.array([0, 1, 1]), np.array([0.3, 0.6, 0.1]), np.array([1.0]))

    # flatten=True
    #
    # Multi-class classification w/ sparse labels
    Input  : labels=[2], predictions=[0.3, 0.6, 0.1]
    Output : (np.array([0]), np.array([0.3]), np.array([1.0])),
             (np.array([0]), np.array([0.6]), np.array([1.0])),
             (np.array([1]), np.array([0.1]), np.array([1.0]))
    # Multi-class/multi-label classification w/ dense labels
    Input  : labels=[0, 0, 1], predictions=[0.3, 0.6, 0.1]
    Output : (np.array([0]), np.array([0.3]), np.array([1.0])),
             (np.array([0]), np.array([0.6]), np.array([1.0])),
             (np.array([1]), np.array([0.1]), np.array([1.0]))

    # sub_key.class_id=[2]
    #
    # Multi-class classification w/ sparse labels
    Input  : labels=[2] predictions=[0.3, 0.6, 0.1]
    Output : (np.array([1]), np.array([0.1]), np.array([1.0]))
    # Multi-class classification w/ dense labels
    Input  : labels=[0, 0, 1] predictions=[0.3, 0.6, 0.1]
    Output : (np.array([1]), np.array([0.1]), np.array([1.0]))

    # sub_key.top_k=2 and aggregation_type is None (i.e. binarization of top 2).
    #
    # Multi-class classification w/ sparse labels
    Input  : labels=[2] predictions=[0.3, 0.6, 0.1]
    Output : (np.array([0, 0, 1]), np.array([0.3, 0.6, -inf]), np.array([1.0]))
    # Multi-class classification w/ dense labels
    Input  : labels=[0, 0, 1] predictions=[0.3, 0.1, 0.6]
    Output : (np.array([0, 0, 1]), np.array([0.3, -inf, 0.6]), np.array([1.0]))

    # sub_key.top_k=2 and aggregation_type is not None (i.e. aggregate top 2).
    #
    # Multi-class classification w/ sparse labels
    Input  : labels=[2] predictions=[0.3, 0.6, 0.1]
    Output : (np.array([0, 1]), np.array([0.3, 0.6]), np.array([1.0]))
    # Multi-class classification w/ dense labels
    Input  : labels=[0, 0, 1] predictions=[0.3, 0.1, 0.6]
    Output : (np.array([0, 0]), np.array([0.3, 0.6]), np.array([1.0]))

    # sub_key.k=2 (i.e. binarization by choosing 2nd largest predicted value).
    #
    # Multi-class classification w/ sparse labels
    Input  : labels=[0] predictions=[0.3, 0.6, 0.1]
    Output : (np.array([1]), np.array([0.3]), np.array([1.0]))
    # Multi-class classification w/ dense labels
    Input  : labels=[0] predictions=[0.3]
    Output : (np.array([0]), np.array([0.3]), np.array([1.0]))

  Args:
    inputs: Standard metric inputs.
    eval_config: Eval config
    model_name: Optional model name (if multi-model evaluation).
    output_name: Optional output name (if multi-output model type).
    sub_key: Optional sub key.
    aggregation_type: Optional aggregation type.
    class_weights: Optional class weights to apply to multi-class / multi-label
      labels and predictions. If used, flatten must also be True.
    example_weighted: True if example weights should be applied.
    fractional_labels: If true, each incoming tuple of (label, prediction, and
      example weight) will be split into two tuples as follows (where l, p, w
      represent the resulting label, prediction, and example weight values): (1)
      l = 0.0, p = prediction, and w = example_weight * (1.0 - label) (2) l =
      1.0, p = prediction, and w = example_weight * label If enabled, an
      exception will be raised if labels are not within [0, 1]. The
      implementation is such that tuples associated with a weight of zero are
      not yielded. This means it is safe to enable fractional_labels even when
      the labels only take on the values of 0.0 or 1.0.
    flatten: True to flatten the final label and prediction outputs so that the
      yielded values are always arrays of size 1. For example, multi-class /
      multi-label outputs would be converted into label and prediction pairs
      that could then be processed by a binary classification metric in order to
      compute a micro average over all classes. If the example weight is not a
      scalar, then they will be flattened as well, otherwise the same example
      weight value will be output for each pair of labels and predictions.
    squeeze: True to squeeze any outputs that have rank > 1. This transforms
      outputs such as np.array([[1]]) to np.array([1]).
    allow_none: True to allow labels or predictions with None values to be
      returned. When used, the values will be returned as empty np.ndarrays. The
      example weight will always be non-empty.
    require_single_example_weight: True to require that the example_weight be a
      single value.

  Yields:
    Tuple of (label, prediction, example_weight).
  """

  def fn_call_str():
    return (f'to_label_prediction_example_weight(inputs={inputs}, '
            f'eval_config={eval_config}, model_name={model_name}, '
            f'output_name={output_name}, sub_key={sub_key}, '
            f'aggregation_type={aggregation_type}, '
            f'class_weights={class_weights}, '
            f'fractional_labels={fractional_labels}, flatten={flatten}, '
            f'squeeze={squeeze}, allow_none={allow_none})')

  def optionally_get_by_keys(value: Any, keys: List[str]) -> Any:

    class NotFound(object):
      pass

    if isinstance(value, Mapping):
      new_value = util.get_by_keys(value, keys, default_value=NotFound())
      if not isinstance(new_value, NotFound):
        # Might be None if that's what is in the dict
        return new_value
    return value

  try:
    prediction_key = ''
    label_key = ''
    if eval_config and eval_config.model_specs:
      for spec in eval_config.model_specs:
        # To maintain consistency between settings where single models are used,
        # always use '' as the model name regardless of whether a name is passed
        spec_name = spec.name if len(eval_config.model_specs) > 1 else ''
        if spec_name == model_name:
          prediction_key = spec.prediction_key
          label_key = spec.label_key
          break

    label = inputs.label
    if label_key:
      # This is to support a custom EvalSavedModel where the labels are a dict
      # but the keys are not output_names.
      label = optionally_get_by_keys(label, [label_key])
    prediction = inputs.prediction
    example_weight = inputs.example_weight
    if model_name:
      if prediction is not None:
        prediction = util.get_by_keys(prediction, [model_name])
      # Labels and weights can optionally be keyed by model name.
      label = optionally_get_by_keys(label, [model_name])
      example_weight = optionally_get_by_keys(example_weight, [model_name])
    if output_name:
      if prediction is not None:
        prediction = util.get_by_keys(prediction, [output_name])
      # Labels and example weights can optionally be keyed by output name.
      label = optionally_get_by_keys(label, [output_name])
      example_weight = optionally_get_by_keys(example_weight, [output_name])

    if not example_weighted or example_weight is None:
      example_weight = np.array(
          1.0, dtype=np.float32
      )  # tf-ranking needs float32

    if isinstance(label, Mapping):
      raise ValueError(
          'unable to prepare label for metric computation because the label is '
          'a dict with unrecognized keys. If a multi-output model was used '
          f'check that an output name was provided in all the relevant '
          'settings (ModelSpec.label_keys, MetricsSpec.output_names, etc): '
          f'label={label}, output_name={output_name}')
    if isinstance(example_weight, Mapping):
      raise ValueError(
          'unable to prepare example_weight for metric computation because the '
          'example_weight is a dict with unrecognized keys. If a multi-output '
          'model was used check that an output name was provided in all the '
          'relevant settings (ModelSpec.example_weight_keys, '
          f'MetricsSpec.output_names, etc): example_weight={example_weight}, '
          f'output_name={output_name}')

    label, prediction = prepare_labels_and_predictions(label, prediction,
                                                       prediction_key)

    if not allow_none:
      for txt, value in zip(('label', 'prediction'), (label, prediction)):
        if value is None:
          raise ValueError(
              f'no value provided for {txt}\n\n'
              'This may be caused by a configuration error (i.e. label, '
              'and/or prediction keys were not specified) or an '
              'error in the pipeline.')

    example_weight = util.to_numpy(example_weight)
    if require_single_example_weight and example_weight.size > 1:
      example_weight = example_weight.flatten()
      if not np.all(example_weight == example_weight[0]):
        raise ValueError(
            'if example_weight size > 0, the values must all be the same: '
            f'example_weight={example_weight}\n\n'
            'This is most likely a configuration error.')
      example_weight = np.array(example_weight[0])

    if sub_key is not None and label is not None and prediction is not None:
      if sub_key.k is not None:
        indices = top_k_indices(sub_key.k, prediction)
        if len(prediction.shape) == 1:
          indices = indices[0]  # 1D
        else:
          # 2D, take kth values
          indices = (indices[0][0::sub_key.k], indices[1][0::sub_key.k])
        if label.shape != prediction.shape:
          label = one_hot(label, prediction)
        label = select_indices(label, indices)
        prediction = select_indices(prediction, indices)
      else:
        if sub_key.top_k is not None:
          # Set all non-top-k predictions to -inf. Note that we do not sort.
          indices = top_k_indices(sub_key.top_k, prediction)
          if aggregation_type is None:
            top_k_predictions = np.full(prediction.shape, float('-inf'))
            top_k_predictions[indices] = prediction[indices]
            prediction = top_k_predictions
          else:
            if label.shape != prediction.shape:
              label = one_hot(label, prediction)
            label = select_indices(label, indices)
            prediction = select_indices(prediction, indices)
        if sub_key.class_id is not None:
          label, prediction = select_class_id(
              sub_key.class_id, label, prediction
          )

    # For consistency, make sure all outputs are arrays (i.e. convert scalars)
    if label is not None and not label.shape:
      label = label.reshape((1,))
    if prediction is not None and not prediction.shape:
      prediction = prediction.reshape((1,))
    if not example_weight.shape:
      example_weight = example_weight.reshape((1,))

    label = label if label is not None else np.array([])
    prediction = prediction if prediction is not None else np.array([])

    flatten_size = prediction.size or label.size
    if flatten:
      if example_weight.size == 1:
        example_weight = np.array(
            [float(example_weight) for i in range(flatten_size)])
      elif example_weight.size != flatten_size:
        raise ValueError(
            'example_weight size does not match the size of labels and '
            'predictions: label={}, prediction={}, example_weight={}'.format(
                label, prediction, example_weight))

    if class_weights:
      if not flatten:
        raise ValueError(
            'class_weights can only be used when flatten is also used: '
            f'class_weights={class_weights}, flatten={flatten}\n\n'
            'This is likely caused by a configuration error (i.e. micro '
            "averaging being applied to metrics that don't support micro "
            'averaging')
      example_weight = np.array([
          example_weight[i] * class_weights[i] if i in class_weights else 0.0
          for i in range(flatten_size)
      ])

    def yield_results(label, prediction, example_weight):
      if (not flatten or (label.size == 0 and prediction.size == 0) or
          (label.size == 1 and prediction.size == 1 and
           example_weight.size == 1)):
        if squeeze:
          yield _squeeze(label), _squeeze(prediction), _squeeze(example_weight)
        else:
          yield label, prediction, example_weight
      elif label.size == 0:
        for p, w in zip(prediction.flatten(), example_weight.flatten()):
          yield label, np.array([p]), np.array([w])
      elif prediction.size == 0:
        for l, w in zip(label.flatten(), example_weight.flatten()):
          yield np.array([l]), prediction, np.array([w])
      elif label.size == prediction.size and label.size == example_weight.size:
        for l, p, w in zip(label.flatten(), prediction.flatten(),
                           example_weight.flatten()):
          yield np.array([l]), np.array([p]), np.array([w])
      elif label.shape[-1] == 1 and prediction.size == example_weight.size:
        label = one_hot(label, prediction)
        for l, p, w in zip(label.flatten(), prediction.flatten(),
                           example_weight.flatten()):
          yield np.array([l]), np.array([p]), np.array([w])
      else:
        raise ValueError(
            'unable to pair labels, predictions, and example weights: '
            f'label={label}, prediction={prediction}, '
            f'example_weight={example_weight}\n\n'
            'This is most likely a configuration error.')

    for result in yield_results(label, prediction, example_weight):
      if fractional_labels and label.size:
        for new_result in _yield_fractional_labels(*result):
          yield new_result
      else:
        yield result
  except Exception as e:
    import sys  # pylint: disable=g-import-not-at-top
    raise type(e)(str(e) + f'\n\n{fn_call_str()}').with_traceback(
        sys.exc_info()[2])

to_standard_metric_inputs

to_standard_metric_inputs(
    extracts: Extracts,
    include_labels: bool = True,
    include_predictions: bool = True,
    include_features: bool = False,
    include_transformed_features: bool = False,
    include_any_feature: bool = False,
    include_attributions: bool = False,
) -> StandardMetricInputs

Verifies extract keys and converts extracts to StandardMetricInputs.

Source code in tensorflow_model_analysis/metrics/metric_util.py

def to_standard_metric_inputs(
    extracts: types.Extracts,
    include_labels: bool = True,
    include_predictions: bool = True,
    include_features: bool = False,
    include_transformed_features: bool = False,
    include_any_feature: bool = False,
    include_attributions: bool = False) -> metric_types.StandardMetricInputs:
  """Verifies extract keys and converts extracts to StandardMetricInputs."""
  if include_labels and constants.LABELS_KEY not in extracts:
    raise ValueError(f'"{constants.LABELS_KEY}" key not found in extracts. '
                     'Check that the configuration is setup properly to '
                     'specify the name of label input and that the proper '
                     'extractor has been configured to extract the labels from '
                     f'the inputs. Existing keys: {extracts.keys()}')
  if include_predictions and constants.PREDICTIONS_KEY not in extracts:
    raise ValueError(f'"{constants.PREDICTIONS_KEY}" key not found in '
                     'extracts. Check that the proper extractor has been '
                     'configured to perform model inference.')
  if include_features and constants.FEATURES_KEY not in extracts:
    raise ValueError(f'"{constants.FEATURES_KEY}" key not found in extracts. '
                     'Check that the proper extractor has been configured to '
                     'extract the features from the inputs. Existing keys: '
                     f'{extracts.keys()}')
  if (include_transformed_features and
      constants.TRANSFORMED_FEATURES_KEY not in extracts):
    raise ValueError(f'"{constants.TRANSFORMED_FEATURES_KEY}" key not found in '
                     'extracts. Check that the proper extractor has been '
                     'configured to extract the transformed features from the '
                     f'inputs. Existing keys: {extracts.keys()}')
  if (include_any_feature and constants.FEATURES_KEY not in extracts and
      constants.TRANSFORMED_FEATURES_KEY not in extracts):
    raise ValueError(
        f'"{constants.FEATURES_KEY}" or {constants.TRANSFORMED_FEATURES_KEY} '
        'key not found in extracts. Check that the proper extractor has been '
        'configured to extract the attributions from the inputs.'
        f'Existing keys: {extracts.keys()}')
  if (include_attributions and constants.ATTRIBUTIONS_KEY not in extracts):
    raise ValueError(f'"{constants.ATTRIBUTIONS_KEY}" key not found in '
                     'extracts. Check that the proper extractor has been '
                     'configured to extract the attributions from the inputs.'
                     f'Existing keys: {extracts.keys()}')
  return metric_types.StandardMetricInputs(extracts)