deepnotebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "## Using this notebook\n",
    "\n",
    "This notebook helps you to analyze datasets and to find interesting and meaningful patterns in the data. If you are only interested in looking at an automated report outlining the most important features of your dataset, you can upload your datafile via the *dataset* variable and run the notebook. Afterwards, you can export the report as HTML and read it in a webbrowser.\n",
    "\n",
    "If you are interested in a more interactive analysis of your data, you can also adapt the parameters of the notebook to suit your needs. Each section conatins several values which can be adapted to your needs. These values are described in the code comments.\n",
    "\n",
    "Finally, if you want to go beyond an automated report and answer your own questions, you can look at the final section of the notebook and use the code examples there to generate your own figures and analysis from the data model.\n",
    "\n",
    "### Reading this report in a webbrowser\n",
    "\n",
    "This report uses several statistical methods and specific phrases and concepts from the domains of statistics and machine learning. Whenever such methods are used, a small \"Explanation\" sign at the side of the report marks a short explanation of the methods and phrases. Clicking it will reveal the explanation.\n",
    "\n",
    "You can toggle the global visibility of these explanations with a button at the top left corner of the report. The code can also be toggled with a button.\n",
    "\n",
    "All graphs are interactive and will display additional content on hover. You can get the exact values of the functions by selecting the assoziated areas in the graph. You can also move the plots around and zoom into interesting parts.\n",
    "\n",
    "### Aknowledgments\n",
    "\n",
    "This notebook is build on the MSPN implementation by Molina et.al. during the course of a bachelor thesis under the supervision of Alejandro Molina and Kristian Kersting at TU Darmstadt. The goal of this framework is to sum product networks for hybrid domains and to highlight important aspects and interesting features of a given dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#from tfspn.SPN import SPN\n",
    "from pprint import PrettyPrinter\n",
    "from IPython.display import Image\n",
    "from IPython.display import display, Markdown\n",
    "from importlib import reload\n",
    "\n",
    "import plotly.plotly as py\n",
    "from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\n",
    "from plotly.graph_objs import *\n",
    "\n",
    "from src.util.text_util import printmd, strip_dataset_name\n",
    "import src.ba_functions as f\n",
    "import src.dn_plot as p\n",
    "import src.dn_text_generation as descr\n",
    "import src.util.data_util as util\n",
    "from src.util.spn_util import get_categoricals\n",
    "\n",
    "from src.util.CSVUtil import learn_piecewise_from_file, load_from_csv\n",
    "\n",
    "init_notebook_mode(connected=True)\n",
    "# pp = PrettyPrinter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# path to the dataset you want to use for training\n",
    "dataset = 'example_data/top20medical.csv'\n",
    "\n",
    "# the minimum number of datapoints that are included in a child of a \n",
    "# sum node\n",
    "min_instances = 50\n",
    "\n",
    "# the parameter which governs how strict the independence test will be\n",
    "# 1 results in all features being evaluated as independent, 0 will \n",
    "# result in no features being acccepted as truly independent\n",
    "independence_threshold = 0.1\n",
    "\n",
    "\n",
    "spn, dictionary = learn_piecewise_from_file(\n",
    "    data_file=dataset, \n",
    "    header=0, \n",
    "    min_instances=min_instances, \n",
    "    independence_threshold=independence_threshold, \n",
    "    feature_file='example_data/top20medical.features')\n",
    "df = pd.read_csv(dataset)\n",
    "context = dictionary['context']\n",
    "context.dataset = strip_dataset_name(dataset)\n",
    "categoricals = get_categoricals(spn, context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to the model pickle file\n",
    "model_path = \"deep_notebooks/models/test.pickle\"\n",
    "\n",
    "# UNCOMMENT THE FOLLOWING LINES TO LOAD A MODEL\n",
    "#spn = pickle.load(open('../myokardinfarkt/spn_save.txt', 'rb'))\n",
    "#df, _, dictionary = load_from_csv('../myokardinfarkt/data/cleaned_pandas.csv', header = 0)\n",
    "#context = pickle.load(open('../myokardinfarkt/context_save.txt', 'rb'))\n",
    "#context.feature_names = ([entry['name']\n",
    "#                                  for entry in dictionary['features']])\n",
    "#dictionary[context] = context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "descr.introduction(context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General statistical evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "descr.data_description(context, df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, the means and standard deviations of each feature are shown. Categorical \n",
    "features do not have a mean and a standard deviation, since they contain no ordering. Instead, \n",
    "the network returns NaN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "descr.means_table(spn, context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following section, the marginal distributions for each feature is shown. This \n",
    "is the distribution of each feature without knowing anything about the other values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "descr.features_shown = 'all'\n",
    "\n",
    "descr.show_feature_marginals(spn, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlations\n",
    "\n",
    "To get a sense of how the features relate to one another, the correlation between \n",
    "them is analyzed in the next section. The correlation denotes how strongly two features are \n",
    "linked. A high correlation (close to 1 or -1) means that two features are very closely related, \n",
    "while a correlation close to 0 means that there is no linear interdependency between the features.\n",
    "\n",
    "The correlation is reported in a colored matrix, where blue denotes a negative and red denotes \n",
    "a positive correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "descr.correlation_threshold = 0.4\n",
    "\n",
    "corr = descr.correlation_description(spn, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The conditional distributions are the probabilities of the features, given \n",
    "a certain instance of a class. The joint probability functions of correlated variables \n",
    "are shown below to allow a more in-depth look into the dependency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "\n",
    "descr.correlation_threshold = 0.2\n",
    "descr.feature_combinations = 'all'\n",
    "descr.show_conditional = True\n",
    "\n",
    "descr.categorical_correlations(spn, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Cluster evaluation\n",
    "\n",
    "To give an impression of the data representation as a whole, the complete network graph is \n",
    "shown below. The model is a tree, with a sum node at its center. The root of the tree is shown \n",
    "in white, while the sum and product nodes are green and blue respectively. Finally, all \n",
    "leaves are represented by red nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#p.plot_graph(spn=spn, fname='deep_notebooks/images/graph.png', context=context)\n",
    "#display(Image(filename='deep_notebooks/images/graph.png', width=400))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data model provides a clustering of the data points into groups in which features are \n",
    "independent. The groups extracted from the data are outlined below together with a short \n",
    "description of the data they cover. Each branch in the model represents one cluster found \n",
    "in the data model.\n",
    "\n",
    "### Description of all clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# possible values: 'all', 'big', int (leading to a random sample), list of nodes to be displayed\n",
    "nodes = f.get_sorted_nodes(spn)\n",
    "\n",
    "reload(descr)\n",
    "descr.nodes = 'all'\n",
    "descr.show_node_graphs = False\n",
    "\n",
    "descr.node_introduction(spn, nodes, context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As stated above, each cluster captures a subgroup of the data. To show what variables are \n",
    "captured by which cluster, the means and variances for each feature and subgroup are plotted below. \n",
    "This highlights where the node has its focus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "descr.features_shown = 'all'\n",
    "descr.mean_threshold = 0.1\n",
    "descr.variance_threshold = 0.1\n",
    "descr.separation_threshold = 0.1\n",
    "\n",
    "separations = descr.show_node_separation(spn, nodes, context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An analysis of the \n",
    "distribution of categorical variables is given below. If a cluster or a group of clusters \n",
    "capture a large fraction of the total likelihood of a categorical instance, they can be \n",
    "interpreted to represent this instance and the associated distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "\n",
    "descr.categoricals = 'all'\n",
    "\n",
    "descr.node_categorical_description(spn, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlations by cluster\n",
    "\n",
    "Finally, since each node captures different interaction between the features, it is \n",
    "interesting to look at the correlations again, this time for the seperate nodes. Shallow \n",
    "nodes are omitted, because the correlation of independent variables is always 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "\n",
    "descr.correlation_threshold = 0.1\n",
    "descr.nodes = 'all'\n",
    "\n",
    "descr.node_correlation(spn, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Predictive data analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(util)\n",
    "numerical_data, categorical_data = util.get_categorical_data(spn, df, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the cluster description, the data model is used to predict data points. To evaluate \n",
    "the performance of the model, the misclassification rate is shown below.\n",
    "\n",
    "The classified data points are used to analyze more advanced patterns within the data, by looking\n",
    "first at the misclassified points, and then at the classification results in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "descr.classify = 'all'\n",
    "\n",
    "misclassified, data_dict = descr.classification(spn, numerical_data, dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, the misclassified examples are explained using the clusters they are most assoiciated with.\n",
    "For each instance, those clusters which form 90 % of the prediction are reported together eith the\n",
    "representatives of these clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMPORTANT: Only set use_shapley to true if you have a really powerful machine\n",
    "reload(descr)\n",
    "reload(p)\n",
    "\n",
    "descr.use_shapley = False\n",
    "descr.shapley_sample_size = 1\n",
    "descr.misclassified_explanations = 1\n",
    "\n",
    "descr.describe_misclassified(spn, dictionary, misclassified, data_dict, numerical_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Information gain through features\n",
    "\n",
    "The following graphs highlight the relative importance of different features for a \n",
    "classification. It can show how different classes are predicted. For continuous and\n",
    "discrete features, a high positvie or negative importance shows that changing this features\n",
    "value positive or negative increases the predictions certainty.\n",
    "\n",
    "For categorical values, positive and negative values highlight whether changing or keeping\n",
    "this categorical value increases or decreasies the predictive certainty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "reload(p)\n",
    "\n",
    "descr.explanation_vector_threshold = 0.2\n",
    "descr.explanation_vector_classes = [20]\n",
    "descr.explanation_vectors_show = 'all'\n",
    "\n",
    "expl_vectors = descr.explanation_vector_description(spn, dictionary, data_dict, categoricals, use_shap=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SHAP explanation\n",
    "\n",
    "The SHAP values for classification show how much influence each feature had on the classification of the datapoint. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from spn.algorithms.Inference import log_likelihood\n",
    "#import numpy as np\n",
    "#\n",
    "#def create_predictor(spn, index, values):\n",
    "#    \n",
    "#    def predict_proba(x):\n",
    "#        all_probs = []\n",
    "#        x[:, index] = np.nan\n",
    "#        normalization = log_likelihood(spn, x)\n",
    "#    \n",
    "#        for v in values:\n",
    "#            x[:, index] = v\n",
    "#            all_probs.append(log_likelihood(spn, x) - normalization)\n",
    "#        print(np.exp(np.array(all_probs)).reshape(x.shape[0], -1))\n",
    "#        return np.exp(np.array(all_probs)).reshape(x.shape[0], -1)\n",
    "#    \n",
    "#    return predict_proba\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import shap\n",
    "#\n",
    "## print the JS visualization code to the notebook\n",
    "#shap.initjs()\n",
    "#\n",
    "## use Kernel SHAP to explain test set predictions\n",
    "#explainer = shap.KernelExplainer(create_predictor(spn, 0, [0,1]), numerical_data)\n",
    "#print('Done 1')\n",
    "#shap_values = explainer.shap_values(numerical_data, nsamples=2)\n",
    "#print('Done 2')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## plot the SHAP values for the Setosa output of the first instance\n",
    "#shap.force_plot(explainer.expected_value[0], shap_values[0][20,:], numerical_data[20,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Conclusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reload(descr)\n",
    "descr.print_conclusion(spn, dictionary, corr, nodes, separations, expl_vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dive into the data\n",
    "\n",
    "Use the Facets Interface to visualize data on your own. You can either load the dataset itself, or show the data as predicted by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load UCI census and convert to json for sending to the visualization\n",
    "import pandas as pd\n",
    "df = pd.read_csv(dataset)\n",
    "jsonstr = df.to_json(orient='records')\n",
    "\n",
    "# Display the Dive visualization for this data\n",
    "from IPython.core.display import display, HTML\n",
    "\n",
    "HTML_TEMPLATE = \"\"\"<link rel=\"import\" href=\"/nbextensions/facets-dist/facets-jupyter.html\">\n",
    "        <facets-dive id=\"elem\" height=\"600\"></facets-dive>\n",
    "        <script>\n",
    "          var data = {jsonstr};\n",
    "          document.querySelector(\"#elem\").data = data;\n",
    "        </script>\"\"\"\n",
    "html = HTML_TEMPLATE.format(jsonstr=jsonstr)\n",
    "display(HTML(html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build your own queries\n",
    "\n",
    "This notebook enables you to add your own analysis to the above. Maybe you are interested in drilling down into specific subclusters of the data, or you want to predict additional datapoint not represented in the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spn.algorithms.Inference import likelihood\n",
    "\n",
    "# get samples to predict\n",
    "data_point = numerical_data[1:2]\n",
    "# get the probability from the models joint probability function\n",
    "proba = likelihood(spn, data_point)\n",
    "\n",
    "\n",
    "printmd(data_point)\n",
    "printmd(likelihood(spn, data_point))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also predict the probability of several data points at the same time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_point = numerical_data[0:3]\n",
    "proba = likelihood(spn, data_point)\n",
    "\n",
    "printmd(data_point)\n",
    "printmd(proba)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src import dn_plot\n",
    "import numpy as np\n",
    "from src.util.spn_util import func_from_spn\n",
    "from spn.algorithms import Inference\n",
    "\n",
    "idx1 = df.columns.get_loc('pe_ecg004')\n",
    "idx2 = df.columns.get_loc('di_dia002')\n",
    "\n",
    "detail = 100\n",
    "\n",
    "x_range = np.linspace(context.domains[idx1][0], context.domains[idx1][1], detail)\n",
    "values = [0,1,2]\n",
    "\n",
    "all_res = []\n",
    "\n",
    "for i in values:\n",
    "    full_data = np.full((detail, df.values.shape[1]), np.nan)\n",
    "    full_data[:, idx1] = x_range\n",
    "    evidence = Inference.likelihood(spn, full_data)\n",
    "    full_data[:, idx2] = i\n",
    "    likelihood = Inference.likelihood(spn, full_data)\n",
    "    all_res.append((likelihood/evidence).squeeze())\n",
    "\n",
    "data = [Scatter(\n",
    "        x=x_range,\n",
    "        y=all_res[i],\n",
    "        mode='lines',\n",
    "    ) for i, _ in enumerate(values)]\n",
    "\n",
    "layout = dict(width=450,\n",
    "              height=450,\n",
    "              xaxis=dict(title=context.feature_names[idx1]),\n",
    "              yaxis=dict(title='Conditional probability')\n",
    "             )\n",
    "\n",
    "\n",
    "iplot({'data': data, 'layout': layout})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src import dn_plot\n",
    "import numpy as np\n",
    "from src.util.spn_util import func_from_spn\n",
    "from spn.algorithms import Inference\n",
    "\n",
    "idx1 = df.columns.get_loc('fi_tre002')\n",
    "idx2 = df.columns.get_loc('di_dia002')\n",
    "\n",
    "detail = 100\n",
    "\n",
    "x_range = np.linspace(context.domains[idx1][0], context.domains[idx1][1], detail)\n",
    "values = [0,1,2]\n",
    "\n",
    "all_res = []\n",
    "\n",
    "for i in values:\n",
    "    full_data = np.full((detail, df.values.shape[1]), np.nan)\n",
    "    full_data[:, idx1] = x_range\n",
    "    evidence = Inference.likelihood(spn, full_data)\n",
    "    full_data[:, idx2] = i\n",
    "    likelihood = Inference.likelihood(spn, full_data)\n",
    "    all_res.append((likelihood/evidence).squeeze())\n",
    "\n",
    "data = [Scatter(\n",
    "        x=x_range,\n",
    "        y=all_res[i],\n",
    "        mode='lines',\n",
    "    ) for i, _ in enumerate(values)]\n",
    "\n",
    "layout = dict(width=450,\n",
    "              height=450,\n",
    "              xaxis=dict(title=context.feature_names[idx1]),\n",
    "              yaxis=dict(title='Conditional probability')\n",
    "             )\n",
    "\n",
    "\n",
    "iplot({'data': data, 'layout': layout})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src import dn_plot\n",
    "import numpy as np\n",
    "from src.util.spn_util import func_from_spn\n",
    "from spn.algorithms import Inference\n",
    "\n",
    "idx1 = df.columns.get_loc('qu_mpa001')\n",
    "idx2 = df.columns.get_loc('di_dia002')\n",
    "\n",
    "detail = 100\n",
    "\n",
    "x_range = np.linspace(context.domains[idx1][0], context.domains[idx1][1], detail)\n",
    "values = [0,1,2]\n",
    "\n",
    "all_res = []\n",
    "\n",
    "for i in values:\n",
    "    full_data = np.full((detail, df.values.shape[1]), np.nan)\n",
    "    full_data[:, idx1] = x_range\n",
    "    evidence = Inference.likelihood(spn, full_data)\n",
    "    full_data[:, idx2] = i\n",
    "    likelihood = Inference.likelihood(spn, full_data)\n",
    "    all_res.append((likelihood/evidence).squeeze())\n",
    "\n",
    "data = [Scatter(\n",
    "        x=x_range,\n",
    "        y=all_res[i],\n",
    "        mode='lines',\n",
    "    ) for i, _ in enumerate(values)]\n",
    "\n",
    "layout = dict(width=450,\n",
    "              height=450,\n",
    "              xaxis=dict(title=context.feature_names[idx1]),\n",
    "              yaxis=dict(title='Conditional probability')\n",
    "             )\n",
    "\n",
    "\n",
    "iplot({'data': data, 'layout': layout})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}