This repository collects different Machine Learning interpretability methods and aims to act as a reference where the user can select the method best suited for their needs. All the methods aim to provide an insight into why a machine learning model has made a given prediction. This is critical because for a model's predictions to be trusted they must be understood.

0. Background
   • Introductory video
1. Interface
2. Explainers
   • Selecting an Interpretability Method
3. Implementation and Notebooks
   • SimplEx
   • Dynamask
   • shap
   • Symbolic Pursuit
4. Explainers By Model Type
   • Tabular Data Model Explainers
   • Time Series model Explainers
   • Unsupervised Model Explainers
   • Individualized Treatment Effect Explainers
5. Generalized and Personalized Explainers
   • Generalized Explainers
   • Personalized Explainers
   • Concept-based Explainers

The Machine Learning (ML) community has produced incredible models for making highly accurate predictions and classifications across many fields. However, uptake of these models into settings outside of the ML community faces a key barrier: Interpretability. If a human cannot understand why a decision has been made by a machine, they cannot be reasonably expected to act on that decision with full confidence, particularly in a high-stakes environment such as medicine. Therefore making the decisions of "Black-Box" models more transparent is of vital importance. For more information see this blog post.

This GitHub repository aims to act as a home for interpretability methods, where the state-of-the-art models can be found for every application. All the linked van der Schaar Lab repositories on this page are pytorch compatible. Pytorch versions of the other methods are available on public libraries, such as captum.

This video is a quick introduction to our Interpretability Suite. It discusses why ML interpretability is so important and shows the array of different methods developed by the van der Schaar Lab that are available on this GitHub page.

The Interpretability Suite provides a common python interface for the following interpretability methods: SimplEx, Dynamask, shap, and Symbolic Pursuit. Each of these methods are also included in our user interface. To guarantee compatibility with the app please create your explainers using the interpretability_suite_app branch.

Figure 1: The Interpretability Suite User Interface Landing page

This user interface not only demonstrates the methods and how they are used on example datasets, but it also gives the user the ability to upload their own explainer to visualize the results. This means that you can save your explainer and give the file to a less python-literate colleague and they can see the results for themselves simply by drag-and-dropping it into the Upload your own Explainer tab.

Figure 2: An example of the Upload your own Explainer tab on the user interface from the SimplEx page

Different model architectures can require different interpretability models, or "Explainers". Below are all the explainers included in this repository, with links to their source code and the papers that introduced them. SimplEx, Dynamask, shap, and Symbolic Pursuit have a common python interface implemented for them for ease of implementation (see Interface above and Implementation and Notebooks below). But any of the other methods can also be implemented by using the code in the GitHub column of the table below.

Figure 3 shows a flowchart to help with the process of selecting the method that is most appropriate for your project.

Figure 3: Interpretability Method selection flowchart.

This repository includes a common python interface for the following interpretability methods: SimplEx, Dynamask, shap, and Symbolic Pursuit. The interface provides the same methods for each of the methods such that you can use the same python methods in your scripts to set up an explainer for each interpretability method. The methods that are:

-  init: Instantiate the class of explainer of your choice.
-  fit: Performs and training for the explainer (This is not required for Shap explainers).
-  explain: Provide the explanation of the data provided.
-  summary_plot: Visualize the explanation.

There are also Notebooks in this GitHub repository to demonstrate how each create the explainer object. These explainers can be saved and uploaded into the Interpretability Suite user interface.

The SimplEx explainer is only compatible with pytorch models. These models must also implement the function latent_representation(). A base class (interpretability.models.base.BlackBox) that your Black-Box model can inherit from is provided, in this package, but this is not a requirement. The notebook to demonstrate the Simplex explainer can be found here for tabular data and here for time series data.

The Dynamask explainer is only compatible with pytorch models for time series data. The notebook to demonstrate the Dynamask explainer can be found here.

The Shap explainer is compatible with pytorch models such as multilayer perceptrons, tree-based models from libraries such as sci-kit learn or xgboost, Kernel models such as those from sci-kit learn, and linear models. The notebook to demonstrate the Shap explainer can be found here. It uses a ShapTreeExplainer to explain the predictions of an XGBoost classifier.

The SimplEx explainer is only compatible with pytorch models and sci-kit learn linear and multilayer perceptron models. The notebook to demonstrate the Symbolic Pursuit explainer can be found here.

The following sections break the methods down by the type of model that they explain.

There are many different static tabular data explainers, giving many different options to choose between. These methods are list below with the inputs required to use them.

The following Explainers work with models for making predictions from time series data.

The following Explainers work with unsupervised clustering ML models, that is to say those without labelled data in the training set.

The following Explainers work with Individualized Treatment Effects (otherwise known as Conditional Average Treatment Effects).

Increased interpretability of a model can be achieved in multiple ways. Generalized methods may provide an explanation for a model's predictions in terms of the features that were important for that decision, e.g. a predicted value y_i was given because feature 1 was high and feature 3 was low for the ith prediction record. Whereas, personalized methods may provide their explanation by showing examples that were important to the prediction, e.g. a predicted value y_i was given because the model had previously seen three records with a similar profile they all had the same label as the predicted value here.

SimplEx is worthy of note in this section as it bridges the gap between these two categories by providing example importances and the features that are important for those examples. It therefore appears in both sections below.

Feature-based explainers have the advantage of specifically telling you the features that are important for a prediction for a given record. These explainers are useful at several points in the development cycle of black-box models. They are useful when debugging a model, because they can reveal places where the model is relying too heavily on fields which you know should be of lesser importance. It is also useful for feature engineering as you can directly see which features are being used the most.

The following explainers provide feature importances for a given prediction.

Example-based explainers show the user example records that the model sees as being similar. They can be useful for debugging in two ways: 1) if the explainer reveals that two examples are being viewed by the model as similar, but the user disagrees, this can be indicative of a model not classifying records correctly; 2) if the explainer states the most similar examples (from the models perspective) are incorrectly classified that also casts doubt on the validity of the predition for the test record.

The following explainers provide example importances for a given prediction.

Concept-based explainers explain the predictive models output in terms of the concepts provided by the user. This has the huge advantage of giving an explanation that is comprised of concepts that the user can understand even though they are not the input features to the model. For example, if there was a classifier that distinguished between horses and zebras one could provide examples of the concept of stripes and see how important stripes are to the predicted values.

The following explainers provide example importances for a given prediction.