Skip to content

TFMA View

tensorflow_model_analysis.view

Initializes TFMA's view rendering api.

Classes

SlicedMetrics

Bases: NamedTuple('SlicedMetrics', [('slice', SliceKeyType), ('metrics', MetricsByOutputName)])

A tuple containing the metrics belonging to a slice.

The metrics are stored in a nested dictionary with the following levels:

  1. output_name: Optional output name associated with metric (for multi-output models). '' by default.
  2. sub_key: Optional sub key associated with metric (for multi-class models). '' by default. See tfma.metrics.SubKey for more info.
  3. metric_name: Name of the metric (auc, accuracy, etc).
  4. metric_value: A dictionary containing the metric's value. See tfma.proto.metrics_for_slice_pb2.MetricValue for more info.

Below is a sample SlicedMetrics:

(
  (('color', 'green')),
  {
    '': {  # default for single-output models
      '': {  # default sub_key for non-multiclass-classification models
        'auc': {
          'doubleValue': 0.7243943810462952
        },
        'accuracy': {
          'doubleValue': 0.6488351225852966
        }
      }
    }
  }
)
ATTRIBUTE DESCRIPTION
slice

A 2-element tuple representing a slice. The first element is the key of a feature (ex: 'color'), and the second element is the value (ex: 'green'). An empty tuple represents an 'overall' slice (i.e. one that encompasses the entire dataset.
metrics

A nested dictionary containing metric names and values.

SlicedPlots

Bases: NamedTuple('SlicedPlots', [('slice', SliceKeyType), ('plot', PlotsByOutputName)])

A tuple containing the plots belonging to a slice.

ATTRIBUTE DESCRIPTION
slice

A 2-element tuple representing a slice. The first element is the key of a feature (ex: 'color'), and the second element is the value (ex: 'green'). An empty tuple represents an 'overall' slice (i.e. one that encompasses the entire dataset.
plot

A dict mapping output_name and sub_key_id to plot data. The data contains histograms and confusion matrices, which can be rendered with the tfma.view.render_plot function.

Functions

render_plot 

render_plot(
    result: EvalResult,
    slicing_spec: Optional[
        Union[SingleSliceSpec, SlicingSpec]
    ] = None,
    output_name: Optional[str] = None,
    class_id: Optional[int] = None,
    top_k: Optional[int] = None,
    k: Optional[int] = None,
    label: Optional[str] = None,
) -> Optional[PlotViewer]

Renders the plot view as widget.

PARAMETER DESCRIPTION
result

An tfma.EvalResult.

TYPE: EvalResult

slicing_spec

The tfma.SlicingSpec to identify the slice. Show overall if unset.

TYPE: Optional[Union[SingleSliceSpec, SlicingSpec]] DEFAULT: None

output_name

A string representing the output name.

TYPE: Optional[str] DEFAULT: None

class_id

A number representing the class id if multi class.

TYPE: Optional[int] DEFAULT: None

top_k

The k used to compute prediction in the top k position.

TYPE: Optional[int] DEFAULT: None

k

The k used to compute prediciton at the kth position.

TYPE: Optional[int] DEFAULT: None

label

A partial label used to match a set of plots in the results.

TYPE: Optional[str] DEFAULT: None

RETURNS DESCRIPTION
Optional[PlotViewer]

A PlotViewer object if in Jupyter notebook; None if in Colab.
Source code in tensorflow_model_analysis/view/widget_view.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
def render_plot(
    result: view_types.EvalResult,
    slicing_spec: Optional[Union[slicer.SingleSliceSpec,
                                 config_pb2.SlicingSpec]] = None,
    output_name: Optional[str] = None,
    class_id: Optional[int] = None,
    top_k: Optional[int] = None,
    k: Optional[int] = None,
    label: Optional[str] = None,
) -> Optional[visualization.PlotViewer]:  # pytype: disable=invalid-annotation
  """Renders the plot view as widget.

  Args:
    result: An tfma.EvalResult.
    slicing_spec: The tfma.SlicingSpec to identify the slice. Show overall if
      unset.
    output_name: A string representing the output name.
    class_id: A number representing the class id if multi class.
    top_k: The k used to compute prediction in the top k position.
    k: The k used to compute prediciton at the kth position.
    label: A partial label used to match a set of plots in the results.

  Returns:
    A PlotViewer object if in Jupyter notebook; None if in Colab.
  """
  if slicing_spec and isinstance(slicing_spec, config_pb2.SlicingSpec):
    slicing_spec = slicer.SingleSliceSpec(spec=slicing_spec)
  slice_spec_to_use = slicing_spec if slicing_spec else slicer.SingleSliceSpec()
  data, cfg = util.get_plot_data_and_config(result.plots, slice_spec_to_use,
                                            output_name, class_id, top_k, k,
                                            label)
  return visualization.render_plot(data, cfg)

render_slicing_attributions 

render_slicing_attributions(
    result: EvalResult,
    slicing_column: Optional[str] = None,
    slicing_spec: Optional[
        Union[SingleSliceSpec, SlicingSpec]
    ] = None,
    metric_name: Optional[str] = None,
    weighted_example_column: Optional[str] = None,
    event_handlers: Optional[
        Callable[[Dict[str, Union[str, float]]], None]
    ] = None,
) -> Optional[SlicingMetricsViewer]

Renders the slicing metrics view as widget.

PARAMETER DESCRIPTION
result

An tfma.EvalResult.

TYPE: EvalResult

slicing_column

The column to slice on.

TYPE: Optional[str] DEFAULT: None

slicing_spec

The tfma.SlicingSpec to filter results. If neither column nor spec is set, show overall.

TYPE: Optional[Union[SingleSliceSpec, SlicingSpec]] DEFAULT: None

metric_name

Name of attributions metric to show attributions for. Optional if only one metric used.

TYPE: Optional[str] DEFAULT: None

weighted_example_column

Override for the weighted example column. This can be used when different weights are applied in different aprts of the model (eg: multi-head).

TYPE: Optional[str] DEFAULT: None

event_handlers

The event handlers

TYPE: Optional[Callable[[Dict[str, Union[str, float]]], None]] DEFAULT: None

RETURNS DESCRIPTION
Optional[SlicingMetricsViewer]

A SlicingMetricsViewer object if in Jupyter notebook; None if in Colab.
Source code in tensorflow_model_analysis/view/widget_view.py 
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
def render_slicing_attributions(
    result: view_types.EvalResult,
    slicing_column: Optional[str] = None,
    slicing_spec: Optional[Union[slicer.SingleSliceSpec,
                                 config_pb2.SlicingSpec]] = None,
    metric_name: Optional[str] = None,
    weighted_example_column: Optional[str] = None,
    event_handlers: Optional[Callable[[Dict[str, Union[str, float]]],
                                      None]] = None,
) -> Optional[visualization.SlicingMetricsViewer]:  # pytype: disable=invalid-annotation
  """Renders the slicing metrics view as widget.

  Args:
    result: An tfma.EvalResult.
    slicing_column: The column to slice on.
    slicing_spec: The tfma.SlicingSpec to filter results. If neither column nor
      spec is set, show overall.
    metric_name: Name of attributions metric to show attributions for. Optional
      if only one metric used.
    weighted_example_column: Override for the weighted example column. This can
      be used when different weights are applied in different aprts of the model
      (eg: multi-head).
    event_handlers: The event handlers

  Returns:
    A SlicingMetricsViewer object if in Jupyter notebook; None if in Colab.
  """
  if slicing_spec and isinstance(slicing_spec, config_pb2.SlicingSpec):
    slicing_spec = slicer.SingleSliceSpec(spec=slicing_spec)
  data = util.get_slicing_metrics(result.attributions, slicing_column,
                                  slicing_spec)
  # Attributions have one additional level of indirection for the metric_name.
  # Filter this out using the metric_name provided.
  for d in data:
    updated_data = {}
    for output_name, per_output_items in d['metrics'].items():  # pytype: disable=attribute-error
      updated_data[output_name] = {}
      for sub_key, per_sub_key_items in per_output_items.items():
        updated_data[output_name][sub_key] = {}
        if metric_name:
          if metric_name not in per_sub_key_items:
            raise ValueError('metric_name={} not found in {}'.format(
                metric_name, per_sub_key_items.keys()))
          updated_data[output_name][sub_key] = per_sub_key_items[metric_name]
        elif len(per_sub_key_items) == 1:
          updated_data[output_name][sub_key] = list(
              per_sub_key_items.values())[0]
        else:
          raise ValueError(
              'metric_name must be one of the following: {}'.format(
                  per_sub_key_items.keys()))
    d['metrics'] = updated_data

  cfg = util.get_slicing_config(result.config, weighted_example_column)

  return visualization.render_slicing_metrics(
      data, cfg, event_handlers=event_handlers)

render_slicing_metrics 

render_slicing_metrics(
    result: EvalResult,
    slicing_column: Optional[str] = None,
    slicing_spec: Optional[
        Union[SingleSliceSpec, SlicingSpec]
    ] = None,
    weighted_example_column: Optional[str] = None,
    event_handlers: Optional[
        Callable[[Dict[str, Union[str, float]]], None]
    ] = None,
) -> Optional[SlicingMetricsViewer]

Renders the slicing metrics view as widget.

PARAMETER DESCRIPTION
result

An tfma.EvalResult.

TYPE: EvalResult

slicing_column

The column to slice on.

TYPE: Optional[str] DEFAULT: None

slicing_spec

The tfma.SlicingSpec to filter results. If neither column nor spec is set, show overall.

TYPE: Optional[Union[SingleSliceSpec, SlicingSpec]] DEFAULT: None

weighted_example_column

Override for the weighted example column. This can be used when different weights are applied in different aprts of the model (eg: multi-head).

TYPE: Optional[str] DEFAULT: None

event_handlers

The event handlers

TYPE: Optional[Callable[[Dict[str, Union[str, float]]], None]] DEFAULT: None

RETURNS DESCRIPTION
Optional[SlicingMetricsViewer]

A SlicingMetricsViewer object if in Jupyter notebook; None if in Colab.
Source code in tensorflow_model_analysis/view/widget_view.py 
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
def render_slicing_metrics(
    result: view_types.EvalResult,
    slicing_column: Optional[str] = None,
    slicing_spec: Optional[Union[slicer.SingleSliceSpec,
                                 config_pb2.SlicingSpec]] = None,
    weighted_example_column: Optional[str] = None,
    event_handlers: Optional[Callable[[Dict[str, Union[str, float]]],
                                      None]] = None,
) -> Optional[visualization.SlicingMetricsViewer]:  # pytype: disable=invalid-annotation
  """Renders the slicing metrics view as widget.

  Args:
    result: An tfma.EvalResult.
    slicing_column: The column to slice on.
    slicing_spec: The tfma.SlicingSpec to filter results. If neither column nor
      spec is set, show overall.
    weighted_example_column: Override for the weighted example column. This can
      be used when different weights are applied in different aprts of the model
      (eg: multi-head).
    event_handlers: The event handlers

  Returns:
    A SlicingMetricsViewer object if in Jupyter notebook; None if in Colab.
  """
  if slicing_spec and isinstance(slicing_spec, config_pb2.SlicingSpec):
    slicing_spec = slicer.SingleSliceSpec(spec=slicing_spec)
  data = util.get_slicing_metrics(result.slicing_metrics, slicing_column,
                                  slicing_spec)
  cfg = util.get_slicing_config(result.config, weighted_example_column)

  return visualization.render_slicing_metrics(
      data, cfg, event_handlers=event_handlers)

render_time_series 

render_time_series(
    results: EvalResults,
    slicing_spec: Optional[
        Union[SingleSliceSpec, SlicingSpec]
    ] = None,
    display_full_path: bool = False,
) -> Optional[TimeSeriesViewer]

Renders the time series view as widget.

PARAMETER DESCRIPTION
results

An tfma.EvalResults.

TYPE: EvalResults

slicing_spec

A tfma.SlicingSpec determining the slice to show time series on. Show overall if not set.

TYPE: Optional[Union[SingleSliceSpec, SlicingSpec]] DEFAULT: None

display_full_path

Whether to display the full path to model / data in the visualization or just show file name.

TYPE: bool DEFAULT: False

RETURNS DESCRIPTION
Optional[TimeSeriesViewer]

A TimeSeriesViewer object if in Jupyter notebook; None if in Colab.
Source code in tensorflow_model_analysis/view/widget_view.py 
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
def render_time_series(
    results: view_types.EvalResults,
    slicing_spec: Optional[Union[slicer.SingleSliceSpec,
                                 config_pb2.SlicingSpec]] = None,
    display_full_path: bool = False
) -> Optional[visualization.TimeSeriesViewer]:  # pytype: disable=invalid-annotation
  """Renders the time series view as widget.

  Args:
    results: An tfma.EvalResults.
    slicing_spec: A tfma.SlicingSpec determining the slice to show time series
      on. Show overall if not set.
    display_full_path: Whether to display the full path to model / data in the
      visualization or just show file name.

  Returns:
    A TimeSeriesViewer object if in Jupyter notebook; None if in Colab.
  """
  if slicing_spec and isinstance(slicing_spec, config_pb2.SlicingSpec):
    slicing_spec = slicer.SingleSliceSpec(spec=slicing_spec)
  slice_spec_to_use = slicing_spec if slicing_spec else slicer.SingleSliceSpec()
  data = util.get_time_series(results, slice_spec_to_use, display_full_path)
  cfg = {'isModelCentric': results.get_mode() == constants.MODEL_CENTRIC_MODE}

  return visualization.render_time_series(data, cfg)