SMS_Snorkel.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SMS_Snorkel.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tauseef1234/Spam_Labeling_Snorkel/blob/main/SMS_Snorkel.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j8qGhoAt-h5-"
      },
      "source": [
        "# Snorkel - Data labeling using weak supervision on SMS Spam Collection Data Set\n",
        "\n",
        "-----\n",
        "\n",
        "## Brief introduction to Snorkel\n",
        "\n",
        "Creating labelled training sets is an expensive and time consuming task for building a machine learning model. The traditional approach to perform this task is to hire a team that would label the data manually.\n",
        "\n",
        "Snorkel provides a framework to programtically label, build and manage training data with [*weak supervision*](https://www.snorkel.org/blog/weak-supervision). Below image shows Snorkel's architecture which consists of three main stages:\n",
        "\n",
        "1. Writing labeling functions (LFs): Labeling function provides an abstracted way to express various heuristic or noisy programmatic labeling strategies. by employing different kinds of weak supervision sources: patterns, heuristics, external knowledge bases, crowdsourced labels, etc.\n",
        "\n",
        "2. Modeling accuracies and correlations: Snorkel creates a generative model based on labeling functions’ correlations, that is, where they agree or disagree.\n",
        "\n",
        "3. Training a discriminative model: While the output is probabilistic labels, the ultimate goal is to train a discriminative model (such as popular ML models) that will be able to generalize beyond the noisy generative model.\n",
        "\n",
        "<img src = 'https://miro.medium.com/max/2000/1*2vim0mfyVRCH_f9cZ90iag.png'>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SqFtzZu155P2"
      },
      "source": [
        "## Objective\n",
        "\n",
        "The objective of this ML project is to train a classifier over text data that can predict whether a text message is spam or not spam. The dataset used for this project has been collected from [papers with code](https://paperswithcode.com/) website. More details about source of the dataset can be found [here](https://paperswithcode.com/dataset/sms-spam-collection-data-set).\n",
        "\n",
        "\n",
        "**Install snorkel**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0gyzec35u3us",
        "outputId": "517a4af5-d0cf-4599-92cc-b4b923ac2b7c"
      },
      "source": [
        "!pip install snorkel==0.9.7"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: snorkel==0.9.7 in /usr/local/lib/python3.7/dist-packages (0.9.7)\n",
            "Requirement already satisfied: scipy<2.0.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (1.4.1)\n",
            "Requirement already satisfied: pandas<2.0.0,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (1.1.5)\n",
            "Requirement already satisfied: numpy<1.20.0,>=1.16.5 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (1.19.5)\n",
            "Requirement already satisfied: networkx<2.4,>=2.2 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (2.3)\n",
            "Requirement already satisfied: munkres>=1.0.6 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (1.1.4)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.33.0 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (4.41.1)\n",
            "Requirement already satisfied: torch<2.0.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (1.9.0+cu102)\n",
            "Requirement already satisfied: tensorboard<2.0.0,>=1.14.0 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (1.15.0)\n",
            "Requirement already satisfied: scikit-learn<0.25.0,>=0.20.2 in /usr/local/lib/python3.7/dist-packages (from snorkel==0.9.7) (0.22.2.post1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas<2.0.0,>=1.0.0->snorkel==0.9.7) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas<2.0.0,>=1.0.0->snorkel==0.9.7) (2.8.1)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.7/dist-packages (from networkx<2.4,>=2.2->snorkel==0.9.7) (4.4.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch<2.0.0,>=1.2.0->snorkel==0.9.7) (3.7.4.3)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (3.17.3)\n",
            "Requirement already satisfied: grpcio>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (1.34.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (1.0.1)\n",
            "Requirement already satisfied: wheel>=0.26; python_version >= \"3\" in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (0.36.2)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (3.3.4)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (1.15.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (57.2.0)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (0.12.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn<0.25.0,>=0.20.2->snorkel==0.9.7) (1.0.1)\n",
            "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (4.6.1)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard<2.0.0,>=1.14.0->snorkel==0.9.7) (3.5.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CDU8D_8Td-IY"
      },
      "source": [
        "**Import Packages**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s_Axz2kLYHH2"
      },
      "source": [
        "# import packages\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import re"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Zr8Z0JGd9LM"
      },
      "source": [
        "\n",
        "\n",
        "## Read the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u2Xnb0ScIdHD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 403
        },
        "outputId": "ecec49d8-f49b-4bb7-a3d2-0b13a3c1b10b"
      },
      "source": [
        "#read spam dataset\n",
        "df_sms = pd.read_csv('https://raw.githubusercontent.com/tauseef1234/Datasets/main/spam.csv',encoding='ISO-8859-1')\n",
        "df_sms.info()\n",
        "df_sms.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 5572 entries, 0 to 5571\n",
            "Data columns (total 5 columns):\n",
            " #   Column      Non-Null Count  Dtype \n",
            "---  ------      --------------  ----- \n",
            " 0   v1          5572 non-null   object\n",
            " 1   v2          5572 non-null   object\n",
            " 2   Unnamed: 2  50 non-null     object\n",
            " 3   Unnamed: 3  12 non-null     object\n",
            " 4   Unnamed: 4  6 non-null      object\n",
            "dtypes: object(5)\n",
            "memory usage: 217.8+ KB\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>v1</th>\n",
              "      <th>v2</th>\n",
              "      <th>Unnamed: 2</th>\n",
              "      <th>Unnamed: 3</th>\n",
              "      <th>Unnamed: 4</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     v1  ... Unnamed: 4\n",
              "0   ham  ...        NaN\n",
              "1   ham  ...        NaN\n",
              "2  spam  ...        NaN\n",
              "3   ham  ...        NaN\n",
              "4   ham  ...        NaN\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eBHNth_78qdN"
      },
      "source": [
        "Some of the messages in the raw file is spread across multiple columns. We append all the text in a single column and then drop the remaining columns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dtRRcPuX8vMp"
      },
      "source": [
        "# replacing nan with blank space\n",
        "cols =['Unnamed: 2','Unnamed: 3','Unnamed: 4']\n",
        "for col in cols:\n",
        "  df_sms[col].replace(np.nan,'',regex=True, inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lTnCFN7RyDGo"
      },
      "source": [
        "# concatnating all columns\n",
        "df_sms['v2']=df_sms['v2'] + df_sms['Unnamed: 2'].astype(str) + df_sms['Unnamed: 3'].astype(str) + df_sms['Unnamed: 4'].astype(str)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_gHwVLDCyWlH"
      },
      "source": [
        "# drop the unnamed columns\n",
        "df_sms.drop(columns=cols,inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jwlnj3Ys_phs"
      },
      "source": [
        "**Prepare the train and test dataset**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EeZWfjyZHGYn"
      },
      "source": [
        "# rename fields\n",
        "df_sms = df_sms.rename(columns={\"v1\": \"label\", \"v2\": \"text\"})\n",
        "# shuffle order\n",
        "df_sms = df_sms.sample(frac=1, random_state=123).reset_index(drop=True)\n",
        "\n",
        "# split the data into train and test datasets\n",
        "from sklearn.model_selection import train_test_split\n",
        "df_train, df_test = train_test_split(\n",
        "        df_sms, test_size=0.2, random_state=123, stratify=df_sms.label\n",
        "    )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sx-EOqO6IDYv",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 421
        },
        "outputId": "d9dce132-8e0b-4fd4-d047-9e9c0d6b8311"
      },
      "source": [
        "# display train and test data\n",
        "print(\"Train\")\n",
        "display(df_train.head())\n",
        "print('Test')\n",
        "df_test.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4114</th>\n",
              "      <td>ham</td>\n",
              "      <td>Double eviction this week - Spiral and Michael...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>179</th>\n",
              "      <td>spam</td>\n",
              "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2332</th>\n",
              "      <td>ham</td>\n",
              "      <td>3 pa but not selected.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3549</th>\n",
              "      <td>ham</td>\n",
              "      <td>Okie</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3240</th>\n",
              "      <td>ham</td>\n",
              "      <td>Anything is valuable in only 2 situations: Fir...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     label                                               text\n",
              "4114   ham  Double eviction this week - Spiral and Michael...\n",
              "179   spam  This is the 2nd time we have tried 2 contact u...\n",
              "2332   ham                             3 pa but not selected.\n",
              "3549   ham                                               Okie\n",
              "3240   ham  Anything is valuable in only 2 situations: Fir..."
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Test\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4565</th>\n",
              "      <td>ham</td>\n",
              "      <td>Alrite jod hows the revision goin? Keris bin d...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2831</th>\n",
              "      <td>ham</td>\n",
              "      <td>When can Ì_ come out?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2489</th>\n",
              "      <td>ham</td>\n",
              "      <td>I'm fine. Hope you are good. Do take care.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3865</th>\n",
              "      <td>ham</td>\n",
              "      <td>Mine here like all fr china then so noisy.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3852</th>\n",
              "      <td>ham</td>\n",
              "      <td>Shant disturb u anymore... Jia you...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     label                                               text\n",
              "4565   ham  Alrite jod hows the revision goin? Keris bin d...\n",
              "2831   ham                              When can Ì_ come out?\n",
              "2489   ham         I'm fine. Hope you are good. Do take care.\n",
              "3865   ham         Mine here like all fr china then so noisy.\n",
              "3852   ham              Shant disturb u anymore... Jia you..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "muw1Zo3HVAiC",
        "outputId": "849d1b0b-d5c7-474b-f77c-fe3582f6086a"
      },
      "source": [
        "# check train and test data dimension\n",
        "print (df_train.info())\n",
        "print (df_test.info())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 4457 entries, 4114 to 3387\n",
            "Data columns (total 2 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   label   4457 non-null   object\n",
            " 1   text    4457 non-null   object\n",
            "dtypes: object(2)\n",
            "memory usage: 104.5+ KB\n",
            "None\n",
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 1115 entries, 4565 to 1929\n",
            "Data columns (total 2 columns):\n",
            " #   Column  Non-Null Count  Dtype \n",
            "---  ------  --------------  ----- \n",
            " 0   label   1115 non-null   object\n",
            " 1   text    1115 non-null   object\n",
            "dtypes: object(2)\n",
            "memory usage: 26.1+ KB\n",
            "None\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uyqFGUGFVP-K",
        "outputId": "ee87ba33-989c-4c63-81f8-da5ea339c22c"
      },
      "source": [
        "df_train['label'].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ham     3859\n",
              "spam     598\n",
              "Name: label, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0TNkZPFuWUkz",
        "outputId": "0b757bf6-481f-407b-919c-77b40c2f693b"
      },
      "source": [
        "df_test['label'].value_counts()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ham     966\n",
              "spam    149\n",
              "Name: label, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d30J7chUQLPZ"
      },
      "source": [
        "# define constants to represent the class labels for spam, ham and abstain\n",
        "ABSTAIN = -1\n",
        "HAM = 0\n",
        "SPAM = 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E5GMNbf7NRBA"
      },
      "source": [
        "# map constant values to the labels 'ham' and 'spam' in the dataset\n",
        "dict_map = {'ham':HAM,'spam':SPAM}\n",
        "Y_train = df_train['label'].map(dict_map).values\n",
        "Y_test = df_test['label'].map(dict_map).values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k9YZtqZj_3sG"
      },
      "source": [
        "## Write Labeling Functions\n",
        "\n",
        "**Pattern-matching LFs**\n",
        "\n",
        "In this step, we write labelling functions using regex pattern matching in order to look for specific syntactical pattern. This is where we will use our \"domain knowledge\" (or in this case mostly common knowledge as most of us recognize patterns of spam and non-spam messages) to write python programs (LFs) that take as input a data point and assign labels using pattern matching.\n",
        "\n",
        "Below are some of the LFs defined using regex pattern matching:\n",
        "\n",
        "1. `smiley_face` - Ham messages are personal which have emoticons included. This LS looks for emoticons\n",
        "2. `regex_pobox` - Spam messages have addresses with 'PO BOX' details\n",
        "3. `regex_tc` -   Promotional spam messages have acronym for 'terms and conditions' like T&C, TnC, TC, T C.\n",
        "4. `spam_numbers` - Spam messages have 5-digit number for texting promotional codes or a 10-digit phone number to call.\n",
        "\n",
        "5. `regex_punc` - Messages shared by many people including family and friends use multiple full stop at the end of sentences. We classify those messages as non-spam."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x2wd5exSa64p"
      },
      "source": [
        "from snorkel.labeling import labeling_function\n",
        "\n",
        "# Ham messages tend to have personal emoticons\n",
        "@labeling_function()\n",
        "def smiley_face(x):\n",
        "    return HAM if re.search(r'(?::|;|=)(?:-)?(?:\\)|\\(|D|P)',x.text) else ABSTAIN\n",
        "\n",
        "# SPAM messages can have post office as text\n",
        "@labeling_function()\n",
        "def regex_pobox(x):\n",
        "    return SPAM if re.search(r\"po.*box\", x.text, flags=re.I) else ABSTAIN\n",
        "\n",
        "# SPAM messages have various acronyms for terms and conditions\n",
        "@labeling_function()\n",
        "def regex_tc(x):\n",
        "    return SPAM if re.search(r\"T.{0,2}C\", x.text) else ABSTAIN\n",
        "\n",
        "# SPAM messages can have 5-digit text numbers or 10-digit phone numbers\n",
        "@labeling_function()\n",
        "def spam_numbers(x):\n",
        "    return SPAM if re.search(r'\\b\\d{5}\\b|\\b\\d{10}\\b',x.text) else ABSTAIN\n",
        "\n",
        "# HAM messages can have multiple full stops\n",
        "@labeling_function()\n",
        "def regex_punc(x):\n",
        "    return HAM if re.search(r'\\.{3,}',x.text) else ABSTAIN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WncJWXbnIVu9"
      },
      "source": [
        "**Keyowrd LFs**\n",
        "\n",
        "Next, we define additional labelling functions using keyword searches to look for specific keywords in a sentence.  Similar to the labeling_function decorator, the LabelingFunction class wraps a Python function (the f parameter), and we can use the resources parameter to pass in keyword arguments to the function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "adkqunikbhGd"
      },
      "source": [
        "from snorkel.labeling import LabelingFunction\n",
        "\n",
        "\n",
        "def keyword_lookup(x, keywords, label):\n",
        "    if any(word in x.text.lower() for word in keywords):\n",
        "        return label\n",
        "    return ABSTAIN\n",
        "\n",
        "def make_keyword_lf(keywords, label=SPAM):\n",
        "    return LabelingFunction(\n",
        "        name=f\"keyword_{keywords[0]}\",\n",
        "        f=keyword_lookup,\n",
        "        resources=dict(keywords=keywords, label=label),\n",
        "    )\n",
        "\n",
        "\"\"\"Spam messages ask users to subscribe to their webpage or channel\"\"\"\n",
        "keyword_subscribe = make_keyword_lf(keywords=[\"subscribe\"])\n",
        "\n",
        "\"\"\"Spam comments post links to webpage or channels\"\"\"\n",
        "keyword_link = make_keyword_lf(keywords=[\"http\",\"www\"])\n",
        "\n",
        "\"\"\"Spam comments make impersonal requests\"\"\"\n",
        "keyword_please = make_keyword_lf(keywords=[\"urgent\", \"private\",\"STOP\", \"satisfaction\", \"guaranteed\", \"FREE\"])\n",
        "\n",
        "\"\"\"Spam comments with calletunes and voicemail\"\"\"\n",
        "keyword_song = make_keyword_lf(keywords=[\"callertunes\",\"voicemail\"])\n",
        "\n",
        "\"\"\"Spam comments actually talk about win and award\"\"\"\n",
        "keyword_win = make_keyword_lf(keywords=[\"reward\", \"points\", \"voucher\", \"bonus\", \"code\", \"gift\", \"claim\", \"prize\", \"expires\", \"award\"])\n",
        "\n",
        "\"\"\"HAM comments have personal references of family and friends\"\"\"\n",
        "keyword_personal =make_keyword_lf(keywords=['bro','mum','papa', 'dad','daddy','mom','mommy', 'father', 'mother'],label=HAM)\n",
        "\n",
        "\"\"\"HAM comments getting delivered via a third party website source\"\"\"\n",
        "keyword_fullonsms =make_keyword_lf(keywords=['fullonsms.com'],label=HAM)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P1_ytksG3LMg"
      },
      "source": [
        "**Heuristic LFs**\n",
        "\n",
        "Rule based labeling functions using heuristics after going through the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OuT5xGgpNnJA"
      },
      "source": [
        "@labeling_function()\n",
        "def short_comment(x):\n",
        "    \"\"\"Ham comments are often short, such as 'cool video!'\"\"\"\n",
        "    return HAM if len(x.text.split()) < 5 else ABSTAIN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jaXv6ijt6-6N"
      },
      "source": [
        "**LFs that use third-party models**\n",
        "\n",
        "Next, we creat a Preprocessor that runs `TextBlob` (a python NLP library) on our text messages, then extracts the polarity and subjectivity scores. Here, we will focus on writing LFs for HAM, since we identified SPAM comments in above LFs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2_my2qzyGApY"
      },
      "source": [
        "from snorkel.preprocess import preprocessor\n",
        "from textblob import TextBlob\n",
        "@preprocessor(memoize=True)\n",
        "def textblob_sentiment(x):\n",
        "    scores = TextBlob(x.text)\n",
        "    x.polarity = scores.sentiment.polarity\n",
        "    x.subjectivity = scores.sentiment.subjectivity\n",
        "    return x \n",
        "\n",
        "@labeling_function(pre=[textblob_sentiment])\n",
        "def textblob_polarity(x):\n",
        "    return HAM if x.polarity > 0.9 else ABSTAIN\n",
        "\n",
        "@labeling_function(pre=[textblob_sentiment])\n",
        "def textblob_subjectivity(x):\n",
        "    return HAM if x.subjectivity >= 0.5 else ABSTAIN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r7mwFVA1EPoY"
      },
      "source": [
        "**LFs with Complex Preprocessor**\n",
        "\n",
        "Here we use the NLP library `spaCy` that can be used to add lemmas, part-of-speech (pos) tags, etc. to each token. Snokel has a prebuilt spaCy preprocessor called `SpacyPreprocessor` which adds a new field to the data point containing a spaCy Doc object."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1qz1pxzBEZ57"
      },
      "source": [
        "from snorkel.preprocess.nlp import SpacyPreprocessor\n",
        "\n",
        "spacy = SpacyPreprocessor(text_field=\"text\", doc_field=\"doc\", memoize=True)\n",
        "\n",
        "\n",
        "@labeling_function(pre=[spacy])\n",
        "def has_person(x):\n",
        "    \"\"\"Ham comments mention specific people and are short.\"\"\"\n",
        "    if len(x.doc) < 20 and any([ent.label_ in [\"PERSON\",\"GPE\"] for ent in x.doc.ents]):\n",
        "        return HAM\n",
        "    else:\n",
        "        return ABSTAIN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kt_5YYrCUiho"
      },
      "source": [
        "from snorkel.labeling.lf.nlp import nlp_labeling_function\n",
        "@nlp_labeling_function()\n",
        "def has_person_nlp(x):\n",
        "    \"\"\"Ham comments mention specific people and are short.\"\"\"\n",
        "    if len(x.doc) < 20 and any([ent.label_ in [\"PERSON\",\"GPE\"] for ent in x.doc.ents]):\n",
        "        return HAM\n",
        "    else:\n",
        "        return ABSTAIN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4nt0NhzxSFo0"
      },
      "source": [
        "**Combining Labeling Functions**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4MBk8iY7sKy1"
      },
      "source": [
        "lfs = [\n",
        "    keyword_subscribe,\n",
        "    keyword_link,\n",
        "    keyword_win,\n",
        "    regex_pobox,\n",
        "    regex_tc,\n",
        "    short_comment,\n",
        "    keyword_please,\n",
        "    keyword_personal,\n",
        "    spam_numbers,\n",
        "    keyword_fullonsms,\n",
        "    regex_punc,\n",
        "    smiley_face,\n",
        "    has_person,\n",
        "    textblob_polarity,\n",
        "    textblob_subjectivity\n",
        "\n",
        "]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bFGF3i9SFcbv"
      },
      "source": [
        "## Building a generalized model using LFs\n",
        "\n",
        "Now, we can apply these LFs to the train and test pandas dataframes to get label matrices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ajo-EqKDvtph",
        "outputId": "f82dd4bd-de15-45b6-d16f-c55eebfda3a0"
      },
      "source": [
        "from snorkel.labeling import PandasLFApplier\n",
        "applier = PandasLFApplier(lfs=lfs)\n",
        "L_train = applier.apply(df=df_train)\n",
        "L_test = applier.apply(df=df_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/tqdm/std.py:658: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
            "  from pandas import Panel\n",
            "100%|██████████| 4457/4457 [01:00<00:00, 73.22it/s]\n",
            "100%|██████████| 1115/1115 [00:13<00:00, 79.82it/s]\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eiE4ySDHHW_Z"
      },
      "source": [
        "Utility function `lf_summary` that provides the following metrics:\n",
        "\n",
        "1. Polarity: The set of unique labels this LF outputs (excluding abstains)\n",
        "2. Coverage: The fraction of the dataset the LF labels\n",
        "3. Overlaps: The fraction of the dataset where this LF and at least one other LF label\n",
        "4. Conflicts: The fraction of the dataset where this LF and at least one other LF label and disagree\n",
        "5. Correct: The number of data points this LF labels correctly (if gold labels are provided)\n",
        "6. Incorrect: The number of data points this LF labels incorrectly (if gold labels are provided)\n",
        "7. Empirical Accuracy: The empirical accuracy of this LF (if gold labels are provided)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 510
        },
        "id": "grjRvEVuytPH",
        "outputId": "6dec5e3d-cefa-4591-efe2-5dc23df0a288"
      },
      "source": [
        "from snorkel.labeling import LFAnalysis\n",
        "LFAnalysis(L=L_train, lfs=lfs).lf_summary(Y_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>j</th>\n",
              "      <th>Polarity</th>\n",
              "      <th>Coverage</th>\n",
              "      <th>Overlaps</th>\n",
              "      <th>Conflicts</th>\n",
              "      <th>Correct</th>\n",
              "      <th>Incorrect</th>\n",
              "      <th>Emp. Acc.</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>keyword_subscribe</th>\n",
              "      <td>0</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.005609</td>\n",
              "      <td>0.005160</td>\n",
              "      <td>0.001795</td>\n",
              "      <td>25</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_http</th>\n",
              "      <td>1</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.019520</td>\n",
              "      <td>0.017725</td>\n",
              "      <td>0.009872</td>\n",
              "      <td>84</td>\n",
              "      <td>3</td>\n",
              "      <td>0.965517</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_reward</th>\n",
              "      <td>2</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.043303</td>\n",
              "      <td>0.037469</td>\n",
              "      <td>0.019969</td>\n",
              "      <td>179</td>\n",
              "      <td>14</td>\n",
              "      <td>0.927461</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>regex_pobox</th>\n",
              "      <td>3</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.016154</td>\n",
              "      <td>0.011891</td>\n",
              "      <td>0.007180</td>\n",
              "      <td>71</td>\n",
              "      <td>1</td>\n",
              "      <td>0.986111</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>regex_tc</th>\n",
              "      <td>4</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.024680</td>\n",
              "      <td>0.022212</td>\n",
              "      <td>0.011891</td>\n",
              "      <td>95</td>\n",
              "      <td>15</td>\n",
              "      <td>0.863636</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>short_comment</th>\n",
              "      <td>5</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.061476</td>\n",
              "      <td>0.034328</td>\n",
              "      <td>0.000224</td>\n",
              "      <td>271</td>\n",
              "      <td>3</td>\n",
              "      <td>0.989051</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_urgent</th>\n",
              "      <td>6</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.021539</td>\n",
              "      <td>0.019295</td>\n",
              "      <td>0.009423</td>\n",
              "      <td>90</td>\n",
              "      <td>6</td>\n",
              "      <td>0.937500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_bro</th>\n",
              "      <td>7</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.030514</td>\n",
              "      <td>0.016379</td>\n",
              "      <td>0.000897</td>\n",
              "      <td>133</td>\n",
              "      <td>3</td>\n",
              "      <td>0.977941</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>spam_numbers</th>\n",
              "      <td>8</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.045546</td>\n",
              "      <td>0.037469</td>\n",
              "      <td>0.023783</td>\n",
              "      <td>202</td>\n",
              "      <td>1</td>\n",
              "      <td>0.995074</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_fullonsms.com</th>\n",
              "      <td>9</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.001346</td>\n",
              "      <td>0.000897</td>\n",
              "      <td>0.000449</td>\n",
              "      <td>6</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>regex_punc</th>\n",
              "      <td>10</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.139556</td>\n",
              "      <td>0.083015</td>\n",
              "      <td>0.000897</td>\n",
              "      <td>614</td>\n",
              "      <td>8</td>\n",
              "      <td>0.987138</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>smiley_face</th>\n",
              "      <td>11</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.067534</td>\n",
              "      <td>0.060803</td>\n",
              "      <td>0.000449</td>\n",
              "      <td>301</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>has_person</th>\n",
              "      <td>12</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.058335</td>\n",
              "      <td>0.031187</td>\n",
              "      <td>0.000897</td>\n",
              "      <td>254</td>\n",
              "      <td>6</td>\n",
              "      <td>0.976923</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>textblob_polarity</th>\n",
              "      <td>13</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.010321</td>\n",
              "      <td>0.009199</td>\n",
              "      <td>0.000897</td>\n",
              "      <td>41</td>\n",
              "      <td>5</td>\n",
              "      <td>0.891304</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>textblob_subjectivity</th>\n",
              "      <td>14</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.426744</td>\n",
              "      <td>0.196769</td>\n",
              "      <td>0.048014</td>\n",
              "      <td>1599</td>\n",
              "      <td>303</td>\n",
              "      <td>0.840694</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                        j Polarity  Coverage  ...  Correct  Incorrect  Emp. Acc.\n",
              "keyword_subscribe       0      [1]  0.005609  ...       25          0   1.000000\n",
              "keyword_http            1      [1]  0.019520  ...       84          3   0.965517\n",
              "keyword_reward          2      [1]  0.043303  ...      179         14   0.927461\n",
              "regex_pobox             3      [1]  0.016154  ...       71          1   0.986111\n",
              "regex_tc                4      [1]  0.024680  ...       95         15   0.863636\n",
              "short_comment           5      [0]  0.061476  ...      271          3   0.989051\n",
              "keyword_urgent          6      [1]  0.021539  ...       90          6   0.937500\n",
              "keyword_bro             7      [0]  0.030514  ...      133          3   0.977941\n",
              "spam_numbers            8      [1]  0.045546  ...      202          1   0.995074\n",
              "keyword_fullonsms.com   9      [0]  0.001346  ...        6          0   1.000000\n",
              "regex_punc             10      [0]  0.139556  ...      614          8   0.987138\n",
              "smiley_face            11      [0]  0.067534  ...      301          0   1.000000\n",
              "has_person             12      [0]  0.058335  ...      254          6   0.976923\n",
              "textblob_polarity      13      [0]  0.010321  ...       41          5   0.891304\n",
              "textblob_subjectivity  14      [0]  0.426744  ...     1599        303   0.840694\n",
              "\n",
              "[15 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 510
        },
        "id": "99yNTSucVYYI",
        "outputId": "0c31aea5-9229-42b1-ac6c-fa49080c5f64"
      },
      "source": [
        "# check summary for the test data\n",
        "LFAnalysis(L=L_test, lfs=lfs).lf_summary(Y_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>j</th>\n",
              "      <th>Polarity</th>\n",
              "      <th>Coverage</th>\n",
              "      <th>Overlaps</th>\n",
              "      <th>Conflicts</th>\n",
              "      <th>Correct</th>\n",
              "      <th>Incorrect</th>\n",
              "      <th>Emp. Acc.</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>keyword_subscribe</th>\n",
              "      <td>0</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.002691</td>\n",
              "      <td>0.002691</td>\n",
              "      <td>0.001794</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_http</th>\n",
              "      <td>1</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.020628</td>\n",
              "      <td>0.020628</td>\n",
              "      <td>0.016143</td>\n",
              "      <td>23</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_reward</th>\n",
              "      <td>2</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.039462</td>\n",
              "      <td>0.030493</td>\n",
              "      <td>0.018834</td>\n",
              "      <td>40</td>\n",
              "      <td>4</td>\n",
              "      <td>0.909091</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>regex_pobox</th>\n",
              "      <td>3</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.014350</td>\n",
              "      <td>0.009865</td>\n",
              "      <td>0.005381</td>\n",
              "      <td>16</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>regex_tc</th>\n",
              "      <td>4</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.026906</td>\n",
              "      <td>0.022422</td>\n",
              "      <td>0.014350</td>\n",
              "      <td>25</td>\n",
              "      <td>5</td>\n",
              "      <td>0.833333</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>short_comment</th>\n",
              "      <td>5</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.071749</td>\n",
              "      <td>0.038565</td>\n",
              "      <td>0.000897</td>\n",
              "      <td>79</td>\n",
              "      <td>1</td>\n",
              "      <td>0.987500</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_urgent</th>\n",
              "      <td>6</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.020628</td>\n",
              "      <td>0.016143</td>\n",
              "      <td>0.009865</td>\n",
              "      <td>21</td>\n",
              "      <td>2</td>\n",
              "      <td>0.913043</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_bro</th>\n",
              "      <td>7</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.030493</td>\n",
              "      <td>0.016143</td>\n",
              "      <td>0.001794</td>\n",
              "      <td>31</td>\n",
              "      <td>3</td>\n",
              "      <td>0.911765</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>spam_numbers</th>\n",
              "      <td>8</td>\n",
              "      <td>[1]</td>\n",
              "      <td>0.043049</td>\n",
              "      <td>0.037668</td>\n",
              "      <td>0.026906</td>\n",
              "      <td>48</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>keyword_fullonsms.com</th>\n",
              "      <td>9</td>\n",
              "      <td>[]</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>regex_punc</th>\n",
              "      <td>10</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.134529</td>\n",
              "      <td>0.082511</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>150</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>smiley_face</th>\n",
              "      <td>11</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.069058</td>\n",
              "      <td>0.058296</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>77</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>has_person</th>\n",
              "      <td>12</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.071749</td>\n",
              "      <td>0.038565</td>\n",
              "      <td>0.001794</td>\n",
              "      <td>80</td>\n",
              "      <td>0</td>\n",
              "      <td>1.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>textblob_polarity</th>\n",
              "      <td>13</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.006278</td>\n",
              "      <td>0.006278</td>\n",
              "      <td>0.002691</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>0.714286</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>textblob_subjectivity</th>\n",
              "      <td>14</td>\n",
              "      <td>[0]</td>\n",
              "      <td>0.409865</td>\n",
              "      <td>0.202691</td>\n",
              "      <td>0.051121</td>\n",
              "      <td>386</td>\n",
              "      <td>71</td>\n",
              "      <td>0.844639</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                        j Polarity  Coverage  ...  Correct  Incorrect  Emp. Acc.\n",
              "keyword_subscribe       0      [1]  0.002691  ...        3          0   1.000000\n",
              "keyword_http            1      [1]  0.020628  ...       23          0   1.000000\n",
              "keyword_reward          2      [1]  0.039462  ...       40          4   0.909091\n",
              "regex_pobox             3      [1]  0.014350  ...       16          0   1.000000\n",
              "regex_tc                4      [1]  0.026906  ...       25          5   0.833333\n",
              "short_comment           5      [0]  0.071749  ...       79          1   0.987500\n",
              "keyword_urgent          6      [1]  0.020628  ...       21          2   0.913043\n",
              "keyword_bro             7      [0]  0.030493  ...       31          3   0.911765\n",
              "spam_numbers            8      [1]  0.043049  ...       48          0   1.000000\n",
              "keyword_fullonsms.com   9       []  0.000000  ...        0          0   0.000000\n",
              "regex_punc             10      [0]  0.134529  ...      150          0   1.000000\n",
              "smiley_face            11      [0]  0.069058  ...       77          0   1.000000\n",
              "has_person             12      [0]  0.071749  ...       80          0   1.000000\n",
              "textblob_polarity      13      [0]  0.006278  ...        5          2   0.714286\n",
              "textblob_subjectivity  14      [0]  0.409865  ...      386         71   0.844639\n",
              "\n",
              "[15 rows x 8 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 697
        },
        "id": "8S7suRe2_ecK",
        "outputId": "adfa55b7-d177-41d7-f7c6-b4a4a800727f"
      },
      "source": [
        "# check some of those messages which the LF regex_tc labelled them as SPAM\n",
        "pd.set_option('display.max_colwidth', None)\n",
        "df_train.loc[L_train[:, 4] == SPAM].sample(20, random_state=3)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>859</th>\n",
              "      <td>spam</td>\n",
              "      <td>Today's Offer! Claim ur å£150 worth of discount vouchers! Text YES to 85023 now! SavaMob, member offers mobile! T Cs 08717898035. å£3.00 Sub. 16 . Unsub reply X</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3831</th>\n",
              "      <td>spam</td>\n",
              "      <td>+449071512431 URGENT! This is the 2nd attempt to contact U!U have WON å£1250 CALL 09071512433 b4 050703 T&amp;CsBCM4235WC1N3XX. callcost 150ppm mobilesvary. maxå£7. 50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1879</th>\n",
              "      <td>spam</td>\n",
              "      <td>If you don't, your prize will go to another customer. T&amp;C at www.t-c.biz 18+ 150p/min Polo Ltd Suite 373 London W1J 6HL Please call back if busy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>410</th>\n",
              "      <td>spam</td>\n",
              "      <td>CDs 4u: Congratulations ur awarded å£500 of CD gift vouchers or å£125 gift guaranteed &amp; Freeentry 2 å£100 wkly draw xt MUSIC to 87066 TnCs www.ldew.com1win150ppmx3age16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4620</th>\n",
              "      <td>spam</td>\n",
              "      <td>The current leading bid is 151. To pause this auction send OUT. Customer Care: 08718726270</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2197</th>\n",
              "      <td>spam</td>\n",
              "      <td>Summers finally here! Fancy a chat or flirt with sexy singles in yr area? To get MATCHED up just reply SUMMER now. Free 2 Join. OptOut txt STOP Help08714742804</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1454</th>\n",
              "      <td>ham</td>\n",
              "      <td>HI ITS KATE CAN U GIVE ME A RING ASAP XXX</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5027</th>\n",
              "      <td>spam</td>\n",
              "      <td>URGENT, IMPORTANT INFORMATION FOR O2 USER. TODAY IS YOUR LUCKY DAY! 2 FIND OUT WHY LOG ONTO HTTP://WWW.URAWINNER.COM THERE IS A FANTASTIC SURPRISE AWAITING FOR YOU</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4997</th>\n",
              "      <td>spam</td>\n",
              "      <td>U have won a nokia 6230 plus a free digital camera. This is what u get when u win our FREE auction. To take part send NOKIA to 83383 now. POBOX114/14TCR/W1 16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5406</th>\n",
              "      <td>spam</td>\n",
              "      <td>Urgent! Please call 09061213237 from a landline. å£5000 cash or a 4* holiday await collection. T &amp;Cs SAE PO Box 177 M227XY. 16+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3298</th>\n",
              "      <td>spam</td>\n",
              "      <td>Think ur smart ? Win å£200 this week in our weekly quiz, text PLAY to 85222 now!T&amp;Cs WinnersClub PO BOX 84, M26 3UZ. 16+. GBP1.50/week</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4187</th>\n",
              "      <td>spam</td>\n",
              "      <td>500 New Mobiles from 2004, MUST GO! Txt: NOKIA to No: 89545 &amp; collect yours today!From ONLY å£1 www.4-tc.biz 2optout 087187262701.50gbp/mtmsg18 TXTAUCTION</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2396</th>\n",
              "      <td>spam</td>\n",
              "      <td>U can WIN å£100 of Music Gift Vouchers every week starting NOW Txt the word DRAW to 87066 TsCs www.Idew.com SkillGame, 1Winaweek, age16. 150ppermessSubscription</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>252</th>\n",
              "      <td>spam</td>\n",
              "      <td>Dear Dave this is your final notice to collect your 4* Tenerife Holiday or #5000 CASH award! Call 09061743806 from landline. TCs SAE Box326 CW25WX 150ppm</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1644</th>\n",
              "      <td>spam</td>\n",
              "      <td>WELL DONE! Your 4* Costa Del Sol Holiday or å£5000 await collection. Call 09050090044 Now toClaim. SAE, TCs, POBox334, Stockport, SK38xh, Costå£1.50/pm, Max10mins</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4841</th>\n",
              "      <td>ham</td>\n",
              "      <td>\\HI BABE UAWAKE?FEELLIKW SHIT.JUSTFOUND OUT VIA ALETTER THATMUM GOTMARRIED 4thNOV.BEHIND OURBACKS åÐ FUCKINNICE!SELFISHDEVIOUSBITCH.ANYWAYIåÕL CALL U\\\"\"</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2037</th>\n",
              "      <td>spam</td>\n",
              "      <td>New Mobiles from 2004, MUST GO! Txt: NOKIA to No: 89545 &amp; collect yours today! From ONLY å£1. www.4-tc.biz 2optout 087187262701.50gbp/mtmsg18 TXTAUCTION.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2570</th>\n",
              "      <td>spam</td>\n",
              "      <td>Urgent! Please call 0906346330. Your ABTA complimentary 4* Spanish Holiday or å£10,000 cash await collection SAE T&amp;Cs BOX 47 PO19 2EZ 150ppm 18+</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2345</th>\n",
              "      <td>spam</td>\n",
              "      <td>URGENT!! Your 4* Costa Del Sol Holiday or å£5000 await collection. Call 09050090044 Now toClaim. SAE, TC s, POBox334, Stockport, SK38xh, Costå£1.50/pm, Max10mins</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5114</th>\n",
              "      <td>spam</td>\n",
              "      <td>You are a å£1000 winner or Guaranteed Caller Prize, this is our Final attempt to contact you! To Claim Call 09071517866 Now! 150ppmPOBox10183BhamB64XE</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     label                                                                                                                                                                       text\n",
              "859   spam           Today's Offer! Claim ur å£150 worth of discount vouchers! Text YES to 85023 now! SavaMob, member offers mobile! T Cs 08717898035. å£3.00 Sub. 16 . Unsub reply X\n",
              "3831  spam        +449071512431 URGENT! This is the 2nd attempt to contact U!U have WON å£1250 CALL 09071512433 b4 050703 T&CsBCM4235WC1N3XX. callcost 150ppm mobilesvary. maxå£7. 50\n",
              "1879  spam                           If you don't, your prize will go to another customer. T&C at www.t-c.biz 18+ 150p/min Polo Ltd Suite 373 London W1J 6HL Please call back if busy\n",
              "410   spam  CDs 4u: Congratulations ur awarded å£500 of CD gift vouchers or å£125 gift guaranteed & Freeentry 2 å£100 wkly draw xt MUSIC to 87066 TnCs www.ldew.com1win150ppmx3age16 \n",
              "4620  spam                                                                                 The current leading bid is 151. To pause this auction send OUT. Customer Care: 08718726270\n",
              "2197  spam            Summers finally here! Fancy a chat or flirt with sexy singles in yr area? To get MATCHED up just reply SUMMER now. Free 2 Join. OptOut txt STOP Help08714742804\n",
              "1454   ham                                                                                                                                  HI ITS KATE CAN U GIVE ME A RING ASAP XXX\n",
              "5027  spam        URGENT, IMPORTANT INFORMATION FOR O2 USER. TODAY IS YOUR LUCKY DAY! 2 FIND OUT WHY LOG ONTO HTTP://WWW.URAWINNER.COM THERE IS A FANTASTIC SURPRISE AWAITING FOR YOU\n",
              "4997  spam             U have won a nokia 6230 plus a free digital camera. This is what u get when u win our FREE auction. To take part send NOKIA to 83383 now. POBOX114/14TCR/W1 16\n",
              "5406  spam                                            Urgent! Please call 09061213237 from a landline. å£5000 cash or a 4* holiday await collection. T &Cs SAE PO Box 177 M227XY. 16+\n",
              "3298  spam                                     Think ur smart ? Win å£200 this week in our weekly quiz, text PLAY to 85222 now!T&Cs WinnersClub PO BOX 84, M26 3UZ. 16+. GBP1.50/week\n",
              "4187  spam                 500 New Mobiles from 2004, MUST GO! Txt: NOKIA to No: 89545 & collect yours today!From ONLY å£1 www.4-tc.biz 2optout 087187262701.50gbp/mtmsg18 TXTAUCTION\n",
              "2396  spam           U can WIN å£100 of Music Gift Vouchers every week starting NOW Txt the word DRAW to 87066 TsCs www.Idew.com SkillGame, 1Winaweek, age16. 150ppermessSubscription\n",
              "252   spam                  Dear Dave this is your final notice to collect your 4* Tenerife Holiday or #5000 CASH award! Call 09061743806 from landline. TCs SAE Box326 CW25WX 150ppm\n",
              "1644  spam         WELL DONE! Your 4* Costa Del Sol Holiday or å£5000 await collection. Call 09050090044 Now toClaim. SAE, TCs, POBox334, Stockport, SK38xh, Costå£1.50/pm, Max10mins\n",
              "4841   ham                   \\HI BABE UAWAKE?FEELLIKW SHIT.JUSTFOUND OUT VIA ALETTER THATMUM GOTMARRIED 4thNOV.BEHIND OURBACKS åÐ FUCKINNICE!SELFISHDEVIOUSBITCH.ANYWAYIåÕL CALL U\\\"\"\n",
              "2037  spam                  New Mobiles from 2004, MUST GO! Txt: NOKIA to No: 89545 & collect yours today! From ONLY å£1. www.4-tc.biz 2optout 087187262701.50gbp/mtmsg18 TXTAUCTION.\n",
              "2570  spam                           Urgent! Please call 0906346330. Your ABTA complimentary 4* Spanish Holiday or å£10,000 cash await collection SAE T&Cs BOX 47 PO19 2EZ 150ppm 18+\n",
              "2345  spam          URGENT!! Your 4* Costa Del Sol Holiday or å£5000 await collection. Call 09050090044 Now toClaim. SAE, TC s, POBox334, Stockport, SK38xh, Costå£1.50/pm, Max10mins\n",
              "5114  spam                     You are a å£1000 winner or Guaranteed Caller Prize, this is our Final attempt to contact you! To Claim Call 09071517866 Now! 150ppmPOBox10183BhamB64XE"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3pImQsCy_2NS"
      },
      "source": [
        "\n",
        "Now, we create a histogram view of how many LF labels the data points in our train set have to get an idea of our total coverage."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "GefLxZCB8n75",
        "outputId": "c69a0736-23ed-40ad-ac8c-fc371a26d862"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "%matplotlib inline\n",
        "\n",
        "def plot_label_frequency(L):\n",
        "    plt.hist((L != ABSTAIN).sum(axis=1), density=True, bins=range(L.shape[1]))\n",
        "    plt.xlabel(\"Number of labels\")\n",
        "    plt.ylabel(\"Fraction of dataset\")\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "plot_label_frequency(L_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAb3klEQVR4nO3de5QedZ3n8feHYMBbEIfMzJKACRgvUbnZxguronKJg5NwZnAMKzOo7GZwQVF0x7B6cDae2QVcHRyMgxEj6qCRi2jPEEUGAccZkTTIxQQjISIkorTAyMg95LN/VLVbeVL9dCV09dPd+bzOeU6qflW/er7JSfLpql/Vr2SbiIiITrv0uoCIiBifEhAREVErAREREbUSEBERUSsBERERtXbtdQGjZa+99vKsWbN6XUZExIRyww03/Nr29LptrQaEpPnAp4ApwPm2zxxmvz8FLgFeYXugbDsdOBF4Eniv7Su6fdesWbMYGBgYzfIjIiY9ST8fbltrASFpCrAMOALYCKyW1G97bcd+zwZOBX5YaZsLLAJeAuwN/LOkF9h+sq16IyJia22OQcwD1tveYPtxYCWwsGa/jwFnAY9W2hYCK20/ZvtnwPryeBERMUbaDIgZwN2V9Y1l2+9IOgTYx/bl29s3IiLa1bO7mCTtAnwS+MBTOMZiSQOSBgYHB0evuIiIaDUgNgH7VNZnlm1Dng28FLhG0p3Aq4B+SX0N+gJge7ntPtt906fXDsJHRMQOajMgVgNzJM2WNJVi0Ll/aKPt39jey/Ys27OA64AF5V1M/cAiSbtJmg3MAa5vsdaIiOjQ2l1MtjdLOgW4guI21xW210haCgzY7u/Sd42ki4C1wGbg5NzBFBExtjRZpvvu6+tznoOIiNg+km6w3Ve3LVNtRERErUkz1cZ4NGtJ5927o+POM49u5bgREVU5g4iIiFoJiIiIqJWAiIiIWgmIiIiolYCIiIhaCYiIiKiVgIiIiFoJiIiIqJWAiIiIWgmIiIiolYCIiIhaCYiIiKiVgIiIiFqZzbXU1syrERETVc4gIiKiVgIiIiJqtRoQkuZLWidpvaQlNdtPknSrpJskfV/S3LJ9lqRHyvabJJ3XZp0REbGt1sYgJE0BlgFHABuB1ZL6ba+t7PYV2+eV+y8APgnML7fdYfugtuqLiIju2jyDmAest73B9uPASmBhdQfbD1ZWnwm4xXoiImI7tBkQM4C7K+sby7atSDpZ0h3A2cB7K5tmS/qRpGslvbbuCyQtljQgaWBwcHA0a4+I2On1fJDa9jLb+wMfAj5SNt8D7Gv7YOA04CuSptX0XW67z3bf9OnTx67oiIidQJsBsQnYp7I+s2wbzkrgGADbj9m+r1y+AbgDeEFLdUZERI02A2I1MEfSbElTgUVAf3UHSXMqq0cDt5ft08tBbiTtB8wBNrRYa0REdGjtLibbmyWdAlwBTAFW2F4jaSkwYLsfOEXS4cATwAPACWX31wFLJT0BbAFOsn1/W7VGRMS2Wp1qw/YqYFVH2xmV5VOH6XcpcGmbtUVERHc9H6SOiIjxKQERERG1EhAREVErAREREbUSEBERUSsBERERtRIQERFRKwERERG1EhAREVErAREREbUSEBERUSsBERERtRIQERFRKwERERG1EhAREVErAREREbUSEBERUSsBERERtVoNCEnzJa2TtF7SkprtJ0m6VdJNkr4vaW5l2+llv3WSjmqzzoiI2FZrASFpCrAMeDMwFziuGgClr9h+me2DgLOBT5Z95wKLgJcA84HPlMeLiIgx0uYZxDxgve0Nth8HVgILqzvYfrCy+kzA5fJCYKXtx2z/DFhfHi8iIsbIri0eewZwd2V9I/DKzp0knQycBkwF3ljpe11H3xk1fRcDiwH23XffUSk6IiIKPR+ktr3M9v7Ah4CPbGff5bb7bPdNnz69nQIjInZSbQbEJmCfyvrMsm04K4FjdrBvRESMsjYDYjUwR9JsSVMpBp37qztImlNZPRq4vVzuBxZJ2k3SbGAOcH2LtUZERIfWxiBsb5Z0CnAFMAVYYXuNpKXAgO1+4BRJhwNPAA8AJ5R910i6CFgLbAZOtv1kW7VGRMS22hykxvYqYFVH2xmV5VO79P0b4G/aqy4iIroZ8RKTpEObtEVExOTSZAzi3IZtERExiQx7iUnSq4HXANMlnVbZNI1iTCEiIiaxbmMQU4Fnlfs8u9L+IHBsm0VFRETvDRsQtq8FrpV0ge2fS3qG7YfHsLaIiOihJmMQe0taC/wEQNKBkj7TblkREdFrTQLiHOAo4D4A2zcDr2uzqIiI6L1GT1LbvrujKQ+tRURMck0elLtb0msAS3oacCpwW7tlRURErzU5gzgJOJliuu1NwEHlekRETGIjnkHY/jXw9jGoJSIixpEmU22cLWmapKdJukrSoKTjx6K4iIjonSaXmI4sXw36FuBO4PnA/2izqIiI6L0mATF0Gepo4GLbv2mxnoiIGCea3MX0T5J+AjwCvFvSdODRdsuKiIheG/EMwvYSikn7+mw/ATwELGy7sIiI6K2mLwzaGzhc0u6Vti+1UE9ERIwTTe5i+ijF+x/OBd4AnA0saHJwSfMlrZO0XtKSmu2nSVor6ZbyDqnnVbY9Kemm8tPf2TciItrVZJD6WOBNwC9tvxM4ENhjpE6SpgDLgDcDc4HjJM3t2O1HFJeuDgAuoQifIY/YPqj8NAqkiIgYPU0C4hHbW4DNkqYB9wL7NOg3D1hve4Ptx4GVdIxd2L66MoX4dcDM5qVHRESbmgTEgKTnAJ8DbgBuBH7QoN8MoDrJ38aybTgnAt+qrO8uaUDSdZKOqesgaXG5z8Dg4GCDkiIioqkmU23893LxPEnfBqbZvmU0iyifzO4DXl9pfp7tTZL2A74r6Vbbd3TUthxYDtDX1+fRrCkiYmfXZJD6qqFl23favqXa1sUmtr4UNbNs6zz+4cCHgQW2H6t816by1w3ANcDBDb4zIiJGybABIWl3Sc8F9pK0p6Tnlp9ZdL9UNGQ1MEfSbElTgUXAVncjSToY+CxFONxbad9T0m7l8l7AocDa7futRUTEU9HtEtNfAu+jeAbiBkBl+4PAp0c6sO3Nkk4BrgCmACtsr5G0FBiw3Q98HHgWcLEkgLvKO5ZeDHxW0haKEDvTdgIiImIMDRsQtj8FfErSe2yfuyMHt70KWNXRdkZl+fBh+v0b8LId+c6IiBgdTQapz5X0UopnGXavtOdJ6oiISWzEgCifpD6MIiBWUTz49n0y1UZExKTW2pPUERExsbX5JHVERExgTWZz7XyS+rc0e5I6IiImsHHxJHVERIw/wwaEpEO6bbN9YzslRUTEeNDtDOIT5a+7U8yTdDPFw3IHAAPAq9stLSIiemnYQWrbb7D9BuAe4BDbfbZfTjEn0jZzKkVExOTS5C6mF9q+dWjF9o8ppsKIiIhJrMldTLdIOh/4h3L97UAGqSMiJrkmAfFO4N3AqeX694C/b62iiIgYF5rc5voo8LflJyIidhJNxiAiImInlICIiIha3d4o9+Xy11OH2yciIiavbmcQL5e0N/CujleOPrd8FWlERExi3QapzwOuAvZj61eOArhsj4iISarbk9R/Z/vFFO+S3s/27MqnUThImi9pnaT1kpbUbD9N0lpJt0i6StLzKttOkHR7+Tlhh353ERGxw5rc5vpuSQcCry2bvtdkNldJU4BlwBHARmC1pH7bayu7/Qjos/2wpHcDZwNvKy9hfZRiDigDN5R9H9ie31xEROy4Ee9ikvRe4ELg98vPhZLe0+DY84D1tjfYfhxYCSys7mD7atsPl6vXATPL5aOAK23fX4bClcD8Jr+hiIgYHU2epP6vwCttPwQg6SyKFwadO0K/GcDdlfWNwCu77H8i8K0ufWd0dpC0GFgMsO+++45QTkREbI8mz0EIeLKy/iRbD1g/ZZKOp7ic9PHt6Wd7eTnLbN/06dNHs6SIiJ1ekzOILwA/lHRZuX4M8PkG/Tax9burZ1IzTbikw4EPA6+3/Vil72Edfa9p8J0RETFKRjyDsP1Jign77i8/77R9ToNjrwbmSJotaSqwCOiv7iDpYOCzwALb91Y2XQEcWT5/sSdwZNkWERFjpMkZBOXrRbfrFaO2N0s6heI/9ikUt8uukbQUGLDdT3FJ6VnAxZIA7rK9wPb9kj5GETIAS23fvz3fHxERT02jgNhRtlcBqzrazqgsH96l7wpgRXvVRUREN5msLyIiaiUgIiKi1oiXmCT9CXAWxUNyKj+2Pa3l2mIYs5Zc3spx7zzz6FaOGxETU5MxiLOBP7Z9W9vFRETE+NHkEtOvEg4RETufJmcQA5K+BnwDGHqQDdtfb62qiIjouSYBMQ14mOJhtSEGEhAREZNYk+m+3zkWhURExPjSZLrvmZIuk3Rv+blU0syR+kVExMTWZJD6CxRzKO1dfv6xbIuIiEmsSUBMt/0F25vLzwVA5taOiJjkmgTEfZKOlzSl/BwP3Nd2YRER0VtNAuJdwJ8BvwTuAY6lmP47IiImsSZ3Mf0cWDAGtURExDgybEBI+ivbZ0s6l+K5h63Yfm+rlUVERE91O4MYml5jYCwKiYiI8WXYgLD9j+Xiw7Yvrm6T9NZWq4qIiJ5rMkh9esO2iIiYRLqNQbwZ+CNghqS/q2yaBmxucnBJ84FPUbyT+nzbZ3Zsfx1wDnAAsMj2JZVtTwK3lqt32c5AeUTEGOo2BvELivGHBcANlfb/AN4/0oElTQGWAUcAG4HVkvptr63sdhfwDuCDNYd4xPZBI31PRES0o9sYxM3AzZIuAx6y/ST87j/+3Rocex6w3vaGst9KYCHwu4CwfWe5bcuO/gYiIqIdTcYgvgM8vbL+dOCfG/SbAdxdWd9YtjW1u6QBSddJOqZuB0mLy30GBgcHt+PQERExkiYBsbvt3w6tlMvPaK+k33me7T7gvwDnSNq/cwfby2332e6bPj3TQ0VEjKYmAfGQpEOGViS9HHikQb9NwD6V9ZllWyO2N5W/bgCuAQ5u2jciIp66Jm+Uex9wsaRfAAL+EHhbg36rgTmSZlMEwyKKs4ERSdqT4vmLxyTtBRwKnN2kb0REjI4mczGtlvQi4IVl0zrbTzTot1nSKcAVFLe5rrC9RtJSYMB2v6RXAJcBewJ/LOl/2X4J8GLgs+Xg9S7AmR13P0VERMuanEFAEQ5zgd2BQyRh+0sjdbK9CljV0XZGZXk1xaWnzn7/BrysYW0REdGCEQNC0keBwygCYhXwZuD7wIgBERERE1eTQepjgTcBv7T9TuBAYI9Wq4qIiJ5rEhCP2N4CbJY0DbiXre9OioiISajJGMSApOcAn6OYcuO3wA9arSoiInqua0BIEvB/bP87cJ6kbwPTbN8yJtVFRETPdA0I25a0ivKOoqG5kyIiYvJrMgZxY/m8QkRE7ESajEG8Ejhe0p3AQxRPU9v2AW0WFhERvdXthUH72r4LOGoM64mIiHGi2xnEN4BDbP9c0qW2/3SsioqIiN7rNgahyvJ+bRcSERHjS7eA8DDLERGxE+h2ielASQ9SnEk8vVyG/z9IPa316iIiome6vZN6ylgWEhER40uT5yAiImInlICIiIhaCYiIiKjVakBImi9pnaT1kpbUbH+dpBslbZZ0bMe2EyTdXn5OaLPOiIjYVmsBIWkKsIziDXRzgeMkze3Y7S7gHcBXOvo+F/goxTQf84CPStqzrVojImJbbZ5BzAPW295g+3FgJbCwuoPtO8upw7d09D0KuNL2/bYfAK4E5rdYa0REdGgzIGYAd1fWN5Zto9ZX0mJJA5IGBgcHd7jQiIjY1oQepLa93Haf7b7p06f3upyIiEmlzYDYxNbvrp5ZtrXdNyIiRkGbAbEamCNptqSpwCKgv2HfK4AjJe1ZDk4fWbZFRMQYaS0gbG8GTqH4j/024CLbayQtlbQAQNIrJG0E3gp8VtKasu/9wMcoQmY1sLRsi4iIMdLkjXI7zPYqYFVH2xmV5dUUl4/q+q4AVrRZX0REDG9CD1JHRER7EhAREVErAREREbUSEBERUSsBERERtRIQERFRKwERERG1EhAREVErAREREbUSEBERUSsBERERtRIQERFRKwERERG1EhAREVErAREREbUSEBERUSsBERERtRIQERFRq9WAkDRf0jpJ6yUtqdm+m6Svldt/KGlW2T5L0iOSbio/57VZZ0REbKu1d1JLmgIsA44ANgKrJfXbXlvZ7UTgAdvPl7QIOAt4W7ntDtsHtVVfRER01+YZxDxgve0Nth8HVgILO/ZZCHyxXL4EeJMktVhTREQ01GZAzADurqxvLNtq97G9GfgN8HvlttmSfiTpWkmvrfsCSYslDUgaGBwcHN3qIyJ2cq1dYnqK7gH2tX2fpJcD35D0EtsPVneyvRxYDtDX1+ce1DmpzFpyeSvHvfPMo1s5bkS0q80ziE3APpX1mWVb7T6SdgX2AO6z/Zjt+wBs3wDcAbygxVojIqJDmwGxGpgjabakqcAioL9jn37ghHL5WOC7ti1pejnIjaT9gDnAhhZrjYiIDq1dYrK9WdIpwBXAFGCF7TWSlgIDtvuBzwNflrQeuJ8iRABeByyV9ASwBTjJ9v1t1RoREdtqdQzC9ipgVUfbGZXlR4G31vS7FLi0zdoiIqK7PEkdERG1EhAREVErAREREbUSEBERUSsBERERtRIQERFRKwERERG1EhAREVErAREREbUSEBERUSsBERERtRIQERFRKwERERG1EhAREVFrvL5yNCaRNl5lmteYRrQvZxAREVErAREREbUSEBERUavVgJA0X9I6SeslLanZvpukr5XbfyhpVmXb6WX7OklHtVlnRERsq7WAkDQFWAa8GZgLHCdpbsduJwIP2H4+8LfAWWXfucAi4CXAfOAz5fEiImKMtHkX0zxgve0NAJJWAguBtZV9FgJ/XS5fAnxaksr2lbYfA34maX15vB+0WG9MIG3cGQW5Oyqiqs2AmAHcXVnfCLxyuH1sb5b0G+D3yvbrOvrO6PwCSYuBxeXqbyWtewr17gX8+in0H0sTqVaYQPXqrIlTa2ki1TuRaoWJVe9TqfV5w22Y0M9B2F4OLB+NY0kasN03Gsdq20SqFSZWvROpVphY9U6kWmFi1dtWrW0OUm8C9qmszyzbaveRtCuwB3Bfw74REdGiNgNiNTBH0mxJUykGnfs79ukHTiiXjwW+a9tl+6LyLqfZwBzg+hZrjYiIDq1dYirHFE4BrgCmACtsr5G0FBiw3Q98HvhyOQh9P0WIUO53EcWA9mbgZNtPtlVraVQuVY2RiVQrTKx6J1KtMLHqnUi1wsSqt5VaVfzAHhERsbU8SR0REbUSEBERUWunD4iRpgMZTyTtI+lqSWslrZF0aq9rGomkKZJ+JOmfel3LSCQ9R9Ilkn4i6TZJr+51TcOR9P7y78CPJX1V0u69rqlK0gpJ90r6caXtuZKulHR7+euevaxxyDC1frz8e3CLpMskPaeXNVbV1VvZ9gFJlrTXaHzXTh0QDacDGU82Ax+wPRd4FXDyOK8X4FTgtl4X0dCngG/bfhFwIOO0bkkzgPcCfbZfSnETyKLeVrWNCyimyalaAlxlew5wVbk+HlzAtrVeCbzU9gHAT4HTx7qoLi5g23qRtA9wJHDXaH3RTh0QVKYDsf04MDQdyLhk+x7bN5bL/0HxH9g2T5iPF5JmAkcD5/e6lpFI2gN4HcWdddh+3Pa/97aqrnYFnl4+P/QM4Bc9rmcrtr9HcWdi1ULgi+XyF4FjxrSoYdTVavs7tjeXq9dRPIs1LgzzZwvFfHZ/BYzanUc7e0DUTQcybv/DrSpnvj0Y+GFvK+nqHIq/sFt6XUgDs4FB4AvlJbHzJT2z10XVsb0J+L8UPyneA/zG9nd6W1Ujf2D7nnL5l8Af9LKY7fAu4Fu9LqIbSQuBTbZvHs3j7uwBMSFJehZwKfA+2w/2up46kt4C3Gv7hl7X0tCuwCHA39s+GHiI8XMJZCvltfuFFKG2N/BMScf3tqrtUz4QO+7vsZf0YYpLuxf2upbhSHoG8D+BM0b72Dt7QEy4KT0kPY0iHC60/fVe19PFocACSXdSXLp7o6R/6G1JXW0ENtoeOiO7hCIwxqPDgZ/ZHrT9BPB14DU9rqmJX0n6TwDlr/f2uJ6uJL0DeAvwdo/vB8b2p/hh4eby39tM4EZJf/hUD7yzB0ST6UDGjXIq9M8Dt9n+ZK/r6cb26bZn2p5F8ef6Xdvj9qdc278E7pb0wrLpTWw9Nf14chfwKknPKP9OvIlxOqDeoTq1zgnAN3tYS1eS5lNcHl1g++Fe19ON7Vtt/77tWeW/t43AIeXf6adkpw6IchBqaDqQ24CLbK/pbVVdHQr8OcVP4zeVnz/qdVGTyHuACyXdAhwE/O8e11OrPMu5BLgRuJXi3/G4mhZC0lcp3t/yQkkbJZ0InAkcIel2irOgM3tZ45Bhav008GzgyvLf2Xk9LbJimHrb+a7xfeYUERG9slOfQURExPASEBERUSsBERERtRIQERFRKwERERG1EhAxoZUzV36isv5BSX89Sse+QNKxo3GsEb7nreXssVd3tM+qm7GzY5/DtnemXEnXSBr1F9zH5JOAiInuMeBPRmt649FSTqLX1InAf7P9hrbqidgRCYiY6DZTPCT2/s4NnWcAkn5b/nqYpGslfVPSBklnSnq7pOsl3Spp/8phDpc0IOmn5fxSQ++4+Lik1eX7Av6yctx/kdRPzVPYko4rj/9jSWeVbWcA/xn4vKSPD/ebLM8m/kXSjeWnOrXGNEmXq3ivyXmSdin7HCnpB+X+F5dzeFWPOaX8M/pxWdc2f4axc9uen3IixqtlwC2Szt6OPgcCL6aYNnkDcL7teSpewvQe4H3lfrMopoXfH7ha0vOBv6CYQfUVknYD/lXS0Gyqh1C8R+Bn1S+TtDdwFvBy4AHgO5KOsb1U0huBD9oe6FLvvcARth+VNAf4KjB0mWgexftMfg58m+KM6hrgI8Dhth+S9CHgNGBp5ZgHATPKd0qgcfRSnBgfEhAx4dl+UNKXKF6i80jDbquHpp6WdAcw9B/8rUD1Us9FtrcAt0vaALyI4qUsB1TOTvYA5gCPA9d3hkPpFcA1tgfL77yQ4v0T32hY79OAT0s6CHgSeEFl2/W2N5TH/SrFGcmjFKHxr8V0TUylmJ6hagOwn6RzgcsrfwYRQAIiJo9zKOYm+kKlbTPlZdTyssvUyrbHKstbKutb2PrfRedcNAYEvMf2FdUNkg6jmCa8De8HfkVx5rMLRQCMVOOVto8b7oC2H5B0IHAUcBLwZxTvPogAMgYRk4Tt+4GLKAZ8h9xJcUkHYAHFT+Hb662SdinHJfYD1lFM7vjucup1JL1AI79c6Hrg9ZL2UvGq2+OAa7ejjj2Ae8qzmT+neM3okHnljMS7AG8Dvk/xFrRDy0tiSHqmpOpZB+XA/i62L6W4HDVepzePHskZREwmn6CYnXfI54BvSrqZ4tr8jvx0fxfFf+7TgJPKMYDzKcYmbiyn2x5khNdn2r5H0hLgaoqf7i+3vT3TXX8GuFTSX7Dt72U1xeyjzy+Pf5ntLSreZ/DVcpwEihD4aaXfDIo36A39oDie3rsc40Bmc42IiFq5xBQREbUSEBERUSsBERERtRIQERFRKwERERG1EhAREVErAREREbX+H94womD73uTUAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KjWM9DAfAhMk"
      },
      "source": [
        "Our next step is to build a generative model based on the LFs. In order to do so, we convert the labels from our LFs into a single column which has noise-aware probablistic model per data point. We can use `MajorityLabelVoter` model that takes a simple majority vote to determine the label for a data point.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J1d_cq0bq_CF"
      },
      "source": [
        "from snorkel.labeling.model import MajorityLabelVoter\n",
        "\n",
        "majority_model = MajorityLabelVoter()\n",
        "preds_train = majority_model.predict(L=L_train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h0TacPd-Jt8g"
      },
      "source": [
        "However, the LFs have varying properties and cannot be treated identically. LFs can be correlated that might results in certain signals being over represented using majority-vote model.\n",
        "\n",
        "Hence, we use `LabelModel` to produce a single set of noise-aware lables, which are probabilistic or confidence-weighted labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2JPNCB1OzB-"
      },
      "source": [
        "from snorkel.labeling.model import LabelModel\n",
        "label_model = LabelModel(cardinality=2,verbose=True) # cardinality is number of classes 2\n",
        "label_model.fit(L_train,n_epochs=1000, log_freq=100,seed=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EElnFcjIOriq",
        "outputId": "680e5590-56da-4e8b-93fe-ba1e356cfdec"
      },
      "source": [
        "majority_acc = majority_model.score(L=L_test, Y=Y_test, tie_break_policy=\"random\")[\n",
        "    \"accuracy\"\n",
        "]\n",
        "print(f\"{'Majority Vote Accuracy:':<25} {majority_acc * 100:.1f}%\")\n",
        "\n",
        "label_model_acc = label_model.score(L=L_test, Y=Y_test, tie_break_policy=\"random\")[\n",
        "    \"accuracy\"\n",
        "]\n",
        "print(f\"{'Label Model Accuracy:':<25} {label_model_acc * 100:.1f}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Majority Vote Accuracy:   77.0%\n",
            "Label Model Accuracy:     75.9%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1EpSXISrUwL5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ce78360e-3809-4407-bf7e-333f4e464946"
      },
      "source": [
        "probs_train = label_model.predict_proba(L=L_train)\n",
        "probs_train"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[5.03393836e-01, 4.96606164e-01],\n",
              "       [2.96629706e-01, 7.03370294e-01],\n",
              "       [5.00000000e-01, 5.00000000e-01],\n",
              "       ...,\n",
              "       [9.99801184e-01, 1.98816027e-04],\n",
              "       [5.61574910e-01, 4.38425090e-01],\n",
              "       [2.55167578e-01, 7.44832422e-01]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49_LP6dBWns7"
      },
      "source": [
        "## Building a discriminator model\n",
        "\n",
        "Now, we have a generative model based on the LFs. However, these models will abstain on the data points our labeling functions don't cover. In the next step, we will use the outputs of the `LabelModel` as training labels to train a discriminative model which can generalize beyond the noisy LFs. This classifier will also only need the text of the comment to make predictions, making it much more suitable for inference over unseen comments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jfSCCSuocNQC"
      },
      "source": [
        "\n",
        "Checking that the labels the LabelModel produces are probabilistic. The following histogram shows the confidences we have that each data point has the label SPAM."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ROpedieWUbN1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "1c16926c-aa44-4b7a-cbdc-b44e2cea7a16"
      },
      "source": [
        "def plot_probabilities_histogram(Y):\n",
        "  plt.hist(Y,bins=10)\n",
        "  plt.xlabel(\"Probability of SPAM\")\n",
        "  plt.ylabel(\"Number of data points\")\n",
        "  plt.show()\n",
        "\n",
        "plot_probabilities_histogram(probs_train[:,1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAYFElEQVR4nO3de7QlZX3m8e8j6HhDwYAEEW01zSR4Q+xBvCwFjQi4BINRIBqBYaUdRaOJKzPERDG6HDWMZMYRNSgESCKI4qUVDCIqGhWh8QKIIi1BhSB0hEEUxQu/+aPeI9umz6mCPnuffXp/P2vttfd+d9WuX/WB85y3qt63UlVIkrSQuy11AZKk6WdYSJJ6GRaSpF6GhSSpl2EhSeq15VIXMA7bbrttrVixYqnLkKRl5aKLLvqPqtpuY59tlmGxYsUK1q5du9RlSNKykuS7833mYShJUi/DQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSr81yBLfUZ8VRZy7Ztq96y7OXbNvSXWXPQpLUy7CQJPUyLCRJvQwLSVIvw0KS1MuwkCT1MiwkSb0MC0lSL8NCktTLsJAk9TIsJEm9DAtJUi/DQpLUy7CQJPUaW1gk2SnJZ5JcluQbSV7Z2h+Q5JwkV7TnbVp7krw9ybokFyfZbeS7Dm3LX5Hk0HHVLEnauHH2LH4JvLqqdgH2AI5MsgtwFHBuVa0Ezm3vAfYFVrbHauBd0IULcDTwBGB34Oi5gJEkTcbYwqKqrq2qr7TXNwPfBHYEDgBOboudDDy3vT4AOKU65wNbJ9kBeBZwTlXdUFU3AucA+4yrbknSHU3knEWSFcDjgC8D21fVte2jHwDbt9c7At8fWe3q1jZfuyRpQsYeFknuC5wBvKqqfjT6WVUVUIu0ndVJ1iZZu379+sX4SklSM9awSHJ3uqD456r6UGu+rh1eoj1f39qvAXYaWf3BrW2+9t9QVcdX1aqqWrXddtst7o5I0owb59VQAU4AvllVx458tAaYu6LpUOCjI+0vbldF7QHc1A5XnQ3snWSbdmJ779YmSZqQLcf43U8G/hi4JMnXWttrgLcApyc5Avgu8IL22VnAfsA64BbgcICquiHJG4EL23JvqKobxli3JGkDYwuLqvpXIPN8/IyNLF/AkfN814nAiYtXnSTpznAEtySpl2EhSeplWEiSehkWkqRehoUkqZdhIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6GRaSpF6GhSSpl2EhSeplWEiSehkWkqRehoUkqZdhIUnqZVhIknoZFpKkXr1hkeTJSe7TXr8oybFJHjr+0iRJ02JIz+JdwC1JHgu8GvgOcMpYq5IkTZUhYfHLqirgAOAdVXUcsNV4y5IkTZMtByxzc5K/BF4EPDXJ3YC7j7csSdI0GdKzOAi4FTiiqn4APBg4ZqxVSZKmypCexZ9V1f+Ye1NV30vyyDHWJEmaMkN6Fs/cSNu+i12IJGl6zduzSPJS4GXAw5NcPPLRVsAXx12YJGl6LHQY6n3AJ4A3A0eNtN9cVTeMtSpJ0lSZNyyq6ibgJuCQJFsA27fl75vkvlX1vQnVKElaYr0nuJO8HHg9cB1wW2su4DHjK0uSNE2GXA31KuA/V9UPx12MJGk6Dbka6vt0h6MkSTNqSM/iSuCzSc6kG5wHQFUdO7aqJElTZUhYfK897tEekqQZ0xsWVfU3kyhEkjS9FhqU97+r6lVJPkZ39dNvqKr9x1qZJGlqLNSz+Mf2/L8mUYgkaXrNezVUVV3Uns8DvgT8sD2+2NoWlOTEJNcnuXSk7fVJrknytfbYb+Szv0yyLsnlSZ410r5Pa1uX5KgNtyNJGr8ht1XdE7gCOA54J/DtJE8d8N0nAftspP3vqmrX9jirbWMX4GDgkW2ddybZoo0cP45u4sJd6EaT7zJg25KkRTTkaqi3AXtX1eUASXYGTgUev9BKVfW5JCsG1nEAcFpV3Qr8W5J1wO7ts3VVdWXb9mlt2csGfq8kaREMGZR397mgAKiqb7Npd8p7eZKL22GqbVrbjnSD/+Zc3drma5ckTdCQsFib5L1J9myP9wBr7+L23gU8AtgVuJau17IokqxOsjbJ2vXr1y/W10qSGBYWL6U77POn7XFZa7vTquq6qvpVVd0GvIfbDzVdA+w0suiDW9t87Rv77uOralVVrdpuu+3uSnmSpHkMGZR3a5J3AOfSzTp7eVX9/K5sLMkOVXVte/sHwNyVUmuA9yU5FngQsBK4AAiwMsnD6ELiYOCP7sq2JUl33ZApyp8NvBv4Dt0v74cleUlVfaJnvVOBPYFtk1wNHA3smWRXukF+VwEvAaiqbyQ5na7X8kvgyKr6VfuelwNnA1sAJ1bVN+7CfkqSNsHQq6H2qqp1AEkeAZxJdxe9eVXVIRtpPmGB5d8EvGkj7WcBZw2oU5I0JkPOWdw8FxTNlcDNY6pHkjSFhvQs1iY5Czid7vDR84ELkxwIUFUfGmN9kqQpMCQs7kl3S9WntffrgXsBz6ELD8NCkjZzQ66GOnwShUiSpteQcxaSpBlnWEiSehkWkqReQ05wzw3MeyTdyW4AquoN4ypKkjRdhtzP4t3AQcAr6EZwPx946JjrkiRNkSGHoZ5UVS8GbqyqvwGeCOw83rIkSdNkSFj8tD3fkuRBwC+AHcZXkiRp2gw5Z/HxJFsDxwBfoRuI996xViVJmipDwuJv2+1Oz0jycbqT3D8bb1mSpGky5DDUl+ZeVNWtVXXTaJskafM3b88iyW/T3e/6XkkeR3clFMD9gHtPoDZJ0pRY6DDUs4DD6G5leuxI+83Aa8ZYkyRpyswbFlV1MnBykudV1RkTrEmSNGWGzDp7hiO4JWm2OYJbktTLEdySpF6O4JYk9XIEtySp15AT3G9sL389grsNzJMkzYiFBuUduMBnVNWHxlOSJGnaLNSzeE57fiDwJODT7f1ewBcBw0KSZsRCg/IOB0jySWCXqrq2vd8BOGki1UmSpsKQq6F2mguK5jrgIWOqR5I0hYZcDXVukrOBU9v7g4BPja8kSdK0GXI11MuT/AHw1NZ0fFV9eLxlSZKmyZCeBS0cDAhJmlFDzllIkmacYSFJ6jVvWCQ5tz2/dXLlSJKm0ULnLHZI8iRg/ySncfttVQGoqq+MtTJJ0tRYKCxeB7yWO95WFbrJBJ8+rqI0O1YcdeZSlyBpgIVGcH8Q+GCS145MJihJmkGDZp1Nsj+3j7P4bFV9fLxlSZKmyZDbqr4ZeCVwWXu8Msn/HHdhkqTpMWRQ3rOBXavqNoAkJwNfBV4zzsIkSdNj6DiLrUde33/ICklOTHJ9kktH2h6Q5JwkV7TnbVp7krw9ybokFyfZbWSdQ9vyVyQ5dGC9kqRFNCQs3gx8NclJrVdxEfCmAeudBOyzQdtRwLlVtRI4t70H2BdY2R6rgXdBFy7A0cATgN2Bo+cCRpI0Ob1hUVWnAnvQ3ezoDOCJVfX+Aet9Drhhg+YDgJPb65OB5460n1Kd84Gt230zngWcU1U3VNWNwDncMYAkSWM2dCLBa4E1i7C97UfujfEDYPv2ekfg+yPLXd3a5mu/gySr6XolPOQh3m5DkhbTks0NVVVFN7hvsb7v+KpaVVWrtttuu8X6WkkSkw+L69rhpbnbs17f2q8BdhpZ7sGtbb52SdIELRgWSbZI8q1F3N4aYO6KpkOBj460v7hdFbUHcFM7XHU2sHeSbdqJ7b1bmyRpghY8Z1FVv0pyeZKHVNX37swXJzkV2BPYNsnVdFc1vQU4PckRwHeBF7TFzwL2A9YBtwCHt+3fkOSNwIVtuTdU1YYnzSVJYzbkBPc2wDeSXAD8ZK6xqvZfaKWqOmSej56xkWULOHKe7zkROHFAnZKkMRkSFq8dexWSpKk2ZCLB85I8FFhZVZ9Kcm9gi/GXJkmaFkMmEvwT4IPA37emHYGPjLMoSdJ0GXLp7JHAk4EfAVTVFcADx1mUJGm6DAmLW6vq53NvkmzJIg6mkyRNvyFhcV6S1wD3SvJM4APAx8ZbliRpmgwJi6OA9cAlwEvoxkT89TiLkiRNlyFXQ93Wpib/Mt3hp8vbuAhJ0ozoDYskzwbeDXwHCPCwJC+pqk+MuzhJ0nQYMijvbcBeVbUOIMkjgDMBw0KSZsSQcxY3zwVFcyVw85jqkSRNoXl7FkkObC/XJjkLOJ3unMXzuX1iP0nSDFjoMNRzRl5fBzytvV4P3GtsFUmSps68YVFVh0+yEEnS9BpyNdTDgFcAK0aX75uiXJK0+RhyNdRHgBPoRm3fNt5yJEnTaEhY/Kyq3j72SiRJU2tIWPyfJEcDnwRunWusqq+MrSpJ0lQZEhaPBv4YeDq3H4aq9l6SNAOGhMXzgYePTlMuSZotQ0ZwXwpsPe5CJEnTa0jPYmvgW0ku5DfPWXjprCTNiCFhcfTYq5AkTbUh97M4bxKFSJKm15AR3Ddz+z237wHcHfhJVd1vnIVJkqbHkJ7FVnOvkwQ4ANhjnEVJkqbLkKuhfq06HwGeNaZ6JElTaMhhqANH3t4NWAX8bGwVSZKmzpCroUbva/FL4Cq6Q1GSpBkx5JyF97WQpBm30G1VX7fAelVVbxxDPZKkKbRQz+InG2m7D3AE8FuAYSFJM2Kh26q+be51kq2AVwKHA6cBb5tvPUnS5mfBcxZJHgD8OfBC4GRgt6q6cRKFSZKmx0LnLI4BDgSOBx5dVT+eWFWSpKmy0KC8VwMPAv4a+PckP2qPm5P8aDLlSZKmwULnLO7U6G5J0ubLQJAk9VqSsEhyVZJLknwtydrW9oAk5yS5oj1v09qT5O1J1iW5OMluS1GzJM2ypexZ7FVVu1bVqvb+KODcqloJnNveA+wLrGyP1cC7Jl6pJM24aToMdQDd5bm05+eOtJ/SZrw9H9g6yQ5LUaAkzaqlCosCPpnkoiSrW9v2VXVte/0DYPv2ekfg+yPrXt3afkOS1UnWJlm7fv36cdUtSTNpyKyz4/CUqromyQOBc5J8a/TDqqokNc+6G1VVx9ONCWHVqlV3al1J0sKWpGdRVde05+uBDwO7A9fNHV5qz9e3xa8BdhpZ/cGtTZI0IRMPiyT3aXNNkeQ+wN7ApcAa4NC22KHAR9vrNcCL21VRewA3jRyukiRNwFIchtoe+HB3O2+2BN5XVf+S5ELg9CRHAN8FXtCWPwvYD1gH3EI3maEkaYImHhZVdSXw2I20/xB4xkbaCzhyAqVJkuYxTZfOSpKmlGEhSeplWEiSehkWkqRehoUkqZdhIUnqZVhIknoZFpKkXoaFJKnXUs06O9VWHHXmkmz3qrc8e0m2K0l97FlIknoZFpKkXoaFJKmX5ywELN15GknLgz0LSVIvw0KS1MuwkCT1MiwkSb0MC0lSL6+GkqQx2NxmgrBnIUnqZVhIknoZFpKkXoaFJKmXYSFJ6mVYSJJ6GRaSpF6GhSSpl2EhSeplWEiSehkWkqRehoUkqZcTCUoau81tUr1ZZFhI2mx5b/nFY1hIM8JfnNoUhoU0Yf7S1nLkCW5JUi/DQpLUa9mERZJ9klyeZF2So5a6HkmaJcsiLJJsARwH7AvsAhySZJelrUqSZsdyOcG9O7Cuqq4ESHIacABw2ZJWtcg88SlpWi2XsNgR+P7I+6uBJ4wukGQ1sLq9/XGSyzdhe9sC/7EJ6y9Hs7bPs7a/4D7PhLx1k/b5ofN9sFzColdVHQ8cvxjflWRtVa1ajO9aLmZtn2dtf8F9nhXj2udlcc4CuAbYaeT9g1ubJGkClktYXAisTPKwJPcADgbWLHFNkjQzlsVhqKr6ZZKXA2cDWwAnVtU3xrjJRTmctczM2j7P2v6C+zwrxrLPqapxfK8kaTOyXA5DSZKWkGEhSeo1s2HRN31Ikv+U5P3t8y8nWTH5KhfXgH3+8ySXJbk4yblJ5r3merkYOk1MkuclqSTL/jLLIfuc5AXtZ/2NJO+bdI2LbcB/2w9J8pkkX23/fe+3FHUuliQnJrk+yaXzfJ4kb2//Hhcn2W2TN1pVM/egO0n+HeDhwD2ArwO7bLDMy4B3t9cHA+9f6ronsM97Afdur186C/vcltsK+BxwPrBqqeuewM95JfBVYJv2/oFLXfcE9vl44KXt9S7AVUtd9ybu81OB3YBL5/l8P+ATQIA9gC9v6jZntWfx6+lDqurnwNz0IaMOAE5urz8IPCNJJljjYuvd56r6TFXd0t6eTzeeZTkb8nMGeCPwVuBnkyxuTIbs858Ax1XVjQBVdf2Ea1xsQ/a5gPu11/cH/n2C9S26qvoccMMCixwAnFKd84Gtk+ywKduc1bDY2PQhO863TFX9ErgJ+K2JVDceQ/Z51BF0f5ksZ7373LrnO1XV5jIx15Cf887Azkm+kOT8JPtMrLrxGLLPrwdelORq4CzgFZMpbcnc2f/fey2LcRaarCQvAlYBT1vqWsYpyd2AY4HDlriUSduS7lDUnnS9x88leXRV/b8lrWq8DgFOqqq3JXki8I9JHlVVty11YcvFrPYshkwf8utlkmxJ13X94USqG49BU6Yk+X3gr4D9q+rWCdU2Ln37vBXwKOCzSa6iO7a7Zpmf5B7yc74aWFNVv6iqfwO+TRcey9WQfT4COB2gqr4E3JNuksHN1aJPkTSrYTFk+pA1wKHt9R8Cn6525miZ6t3nJI8D/p4uKJb7cWzo2eequqmqtq2qFVW1gu48zf5VtXZpyl0UQ/7b/ghdr4Ik29IdlrpykkUusiH7/D3gGQBJfo8uLNZPtMrJWgO8uF0VtQdwU1VduylfOJOHoWqe6UOSvAFYW1VrgBPouqrr6E4kHbx0FW+6gft8DHBf4APtXP73qmr/JSt6Ew3c583KwH0+G9g7yWXAr4C/qKpl22seuM+vBt6T5M/oTnYftpz/+EtyKl3gb9vOwxwN3B2gqt5Nd15mP2AdcAtw+CZvcxn/e0mSJmRWD0NJku4Ew0KS1MuwkCT1MiwkSb0MC0lSL8NCy1KSXyX5WpJLk3wgyb3vxLqHJXnHndzej+dpf0MbyEiSz84N6EtyVpKt2+Nld2ZbPXUc02aKPWaD9u2TfDzJ19tssme19hVJftr+rS5L8u42cp0k2yb5RZL/tsF3XZXk8xu0fW2+GU41GwwLLVc/rapdq+pRwM+BDX/hTWQMUVW9rqo+tZH2/dr0GVvTzWC8WFYDj6mqv9ig/Q3AOVX12KraBRidpvs7VbUr8Bi6GVef29qfTzcQ8ZCNbGerJHMzGPzeItavZcqw0Obg88DvJNkzyeeTrAEuS3LPJP+Q5JJ2H4O9RtbZqfUErkhy9Fxjko8kuaj99b56dCNJ/q61n5tku9Z2UpI/3LCg9tf5tsBbgEe0v8yPSXJKkueOLPfPSQ7YYN20ZS9ttR/U2tfQDZq8aK5txA5003gAUFUXb1hTmxDzi8DvtKZD6Aar7ZhkwxmGTwcOGlnu1A2/T7PFsNCy1noQ+wKXtKbdgFdW1c7AkUBV1aPpfuGdnOSebbndgefR/bX9/JH5oP5rVT2ebiLFP00yN9PwfehGAz8SOI9uxOwQR9H+sm+9gRNoExcmuT/wJGDDGW8PBHYFHgv8PnBMkh3aaPq5HtX7N1jnOOCEdDf4+askD9rIv9W96aa8uKT1Gnaoqgv4zWCYc0arA+A5wMcG7q82U4aFlqt7JfkasJZu3p8TWvsFbXI8gKcA/wRQVd8Cvks3DxJ0h2x+WFU/BT7UloUuIL5Od3hmJ26fYO82YO4X9D+NLH+nVNV5dPMYbUcXYGe0v/hHPQU4tap+VVXX0YXTf+n53rPpbv7zHuB3ga/O9X5oPRvgC8CZVfUJunA4vX1+Gnc8FPVD4MYkBwPfpJsyQjNsJueG0mbhp+04/K+1+ax+MnD9Dee5qSR70v0l/8SquiXJZ+kmnBuy/p1xCvAiuvnGNnnOnjlVdQPwPuB9ST5Odze1i7j9nMWoQ4DfTvLC9v5BSVZW1RUjy7yfrsdy2GLVqOXLnoU2Z58HXgiQZGfgIcDl7bNnJnlAknvRnfD9At009De2oPhduinL59yNbvZhgD8C/nVgDTfTTYU+6iTgVQBVddk8dR+UZIvWO3gqcMFCG0ny9LkrwpJsBTyCrse1sWV3Bu5bVTuOzLj7Zu7Yu/gw8Ld0E/RpxhkW2py9E7hbkkvo/ko+bOQeHRfQHZe/mO5Q0FrgX4Atk3yT7sT0+SPf9RNg93b56NPprj7q1WZz/UI7WX1Ma7uO7tDOP8yz2odbXV8HPg3896r6Qc+mHg+sTXIx8CXgvVV14TzLHtK2MeoMNgiLqrq5qt7ablWqGeess9KEtR7AJcBuVXXTUtcjDWHPQpqgNoDvm8D/NSi0nNizkCT1smchSeplWEiSehkWkqRehoUkqZdhIUnq9f8B4WA9TBQG+UcAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f4DlXkxK91jQ"
      },
      "source": [
        "**Filtering out unlabeled data points**\n",
        "\n",
        "As we saw earlier, some of the data points in our train set received no labels from any of our LFs. These data points convey no supervision signal and tend to hurt performance, so we filter them out before training using a built-in utility."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qcd7A2-QQZov"
      },
      "source": [
        "from snorkel.labeling import filter_unlabeled_dataframe\n",
        "\n",
        "df_train_filtered, probs_train_filtered = filter_unlabeled_dataframe(\n",
        "    X=df_train, y=probs_train, L=L_train\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NvloviDrUvPy",
        "outputId": "a701616d-9d1b-405e-885d-bcaa3f4bf056"
      },
      "source": [
        "probs_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(4457, 2)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QG58sv11Qbue",
        "outputId": "1c951103-c66a-4253-81f8-5def79342766"
      },
      "source": [
        "print(f\"{'train filtered:':<20}{df_train_filtered.shape}\")\n",
        "print(f\"{'prob filtered:':<20}{probs_train_filtered.shape}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train filtered:     (2845, 2)\n",
            "prob filtered:      (2845, 2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMqFIwLmcm1Z"
      },
      "source": [
        "**Training a Classifier**\n",
        "\n",
        "In this step, the output of the `LabelModel` is used to train a classifier. The discriminative model learns feature representation of our labeling functions and this makes it better able to generalize to unseen data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j6J2ibTRh0Ca"
      },
      "source": [
        "Featurization \n",
        "\n",
        "Here, we use a simple \"bag of n-grams\" feature representation: each data point is represented by a one-hot vector marking which words or 2-word combinations are present in the comment text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HP1vY0xzRoOs"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "\n",
        "vectorizer = CountVectorizer(ngram_range=(1, 5))\n",
        "X_train = vectorizer.fit_transform(df_train_filtered.text.tolist())\n",
        "X_test = vectorizer.transform(df_test.text.tolist())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ypx6bcF29g1y",
        "outputId": "3b417c6b-430c-4397-c1af-2f87ed77816c"
      },
      "source": [
        "X_train.shape"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2845, 119780)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oa1cCoZtiHFn"
      },
      "source": [
        "**Logisitic Regression Classifier**\n",
        "\n",
        "First,  we replace each label distribution with the label of the class that has the maximum probability using helper function `probs_to_pred`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KzY1YseqYBzS"
      },
      "source": [
        "from snorkel.utils import probs_to_preds\n",
        "\n",
        "preds_train_filtered = probs_to_preds(probs=probs_train_filtered)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jwlKL1_Fio4n"
      },
      "source": [
        "We use these labels to train our classifier."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mxkTqquMapA_",
        "outputId": "4347c900-4fdd-4db7-945b-01e25d62d93d"
      },
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import classification_report, confusion_matrix\n",
        "\n",
        "sklearn_model = LogisticRegression(C=1e3, solver=\"liblinear\")\n",
        "sklearn_model.fit(X=X_train, y=preds_train_filtered)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression(C=1000.0, class_weight=None, dual=False, fit_intercept=True,\n",
              "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
              "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
              "                   random_state=None, solver='liblinear', tol=0.0001, verbose=0,\n",
              "                   warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rUMjBRZSarrS",
        "outputId": "585b7961-9f80-4520-d061-a5f1ae82f8ed"
      },
      "source": [
        "# checking accuracy\n",
        "print(f\"Test Accuracy: {sklearn_model.score(X=X_test, y=Y_test) * 100:.1f}%\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Accuracy: 94.9%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        },
        "id": "wdzxBI0MPdwa",
        "outputId": "f35b8c4e-a11c-4bd7-b781-1c513e4fbadc"
      },
      "source": [
        "from sklearn import metrics\n",
        "\n",
        "#predicted probablities for label 1\n",
        "y_pred_prob = sklearn_model.predict_proba(X_test)[:,1]\n",
        "\n",
        "fpr, tpr, thresholds = metrics.roc_curve(Y_test,y_pred_prob,pos_label=1)\n",
        "plt.plot(fpr, tpr, 'b')\n",
        "plt.plot([0,1],[0,1], 'r--')\n",
        "plt.xlabel(\"False Positive Rate\", fontsize=12)\n",
        "plt.ylabel(\"True Positive Rate\", fontsize=12)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Text(0, 0.5, 'True Positive Rate')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEJCAYAAAB7UTvrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU9fX/8dcBRERRVNCvIkUFFWyRLPaCHQuiMRp7CYK9xF4So0SxYkFBKRZExUIsRInoz4bRoKIoioqsKE2UEkABBRbO74/P3WVYt8zuzty7M/N+Ph7z2Dt37s49d1n2zKfc8zF3R0REBKBB0gGIiEj9oaQgIiJllBRERKSMkoKIiJRRUhARkTKNkg6gLlq0aOHt2rVLOgwRkZzy0UcfzXP3lhW9ltNJoV27dowfPz7pMEREcoqZTavsNXUfiYhIGSUFEREpo6QgIiJllBRERKSMkoKIiJSJJSmY2cNmNsfMPq/kdTOz/mZWbGYTzaxzHHGJiMia4mopPAp0q+L1w4AO0aM38EAMMYmISDmx3Kfg7mPNrF0Vh/QAHvNQx3ucmTU3s83cfXYc8YmIxOWtt+CNN2r//WstX8K6S+eyz6nt6NIlY2GVqS83r7UCZqQ8nxnt+01SMLPehNYEbdq0iSU4EZF0zJ8Ps2ZVfcxFF8Fnn4FZzd9/f3+DwfRiERvwwXbj6dIl85099SUppM3dBwODAYqKirRCkIjUG7vuClOnVn/cscfCyJE1eOOFC+GKK2DoUGjfHobeTef9stP7X1+SwiygdcrzLaJ9IiI1dtFF8Mor8Z/322/h8MOhZ8+qj9t99xq86cqVsOeeMHkyXHkl3HADrLNOXcKsUn1JCqOAC8zsKWA3YJHGE0Ryx6pVcPfdMG9e0pEEjz0GzZuHv6Vx6tIlJKTddsvAm82fDxttBA0bws03Q+vWUFSUgTeuWixJwcxGAF2BFmY2E/g7sBaAuz8IjAYOB4qBpcCZccQlIulbsgS++qri12bOhMsvD9uNG8cXU1WuvTZ8sM457vDEE3DxxXDrrdCrFxxzTGynj2v20YnVvO7A+XHEIiK106sXjBhR9TEvvAA9esQTT16aMQPOOQdGjw59THvtFXsI9aX7SESq8OqrcO65oXs5KT/+CB06QL9+Fb/epAnsv3+8MeWVESPg7LPDP/I998AFF4Suo5gpKYjUc2++CX36hFktp54KDRIsTnPEEdC9e3Lnz2sbbhgGIwYPhi23TCwMJQUpWLNmhdZ6fXfllfDRR7DLLvDoo8kmBcmgkpIwOr98OVx3HXTrBoceWrsbGDJISUEK1u9/H7pEckG3bvDvfycdhWTMp5+GeasffQTHHx8Gl80STwigpCA5bPRouPBC+PXX2n3/jz/CH/9Y/Zzy+mDnnZOOQDJi2TK46aYwq2ijjeDZZ8OdbPUgGZRSUpCs6t8fvvkmO+/9+OPQsiUccEDtvr9BgzCWt+OOmY1LpFJTpsBtt8FJJ8Fdd8HGGycd0W8oKQgrV8L48aFrM5NWrAhTrZs0CY9M23RTGDUKttkm8+8tkjGLF8OLL8LJJ8MOO4SbPbbaKumoKqWkIDzzTPjgki133BE+kYsUnNdeg969Ydo06NwZOnas1wkBlBQKyooVsMceMH36mvt/+SV8feaZ0M2ZSY0ahXOKFJQFC8It3g8/HJqyb78dEkIOUFJIyNtvw9NPx3vOX38Nkx322gt22mnN1zbfPAy61qPxLpHctHJl+E/29ddwzTVw/fXZ6T/NEiWFGM2aFQodQvg9GTcu85/Mq7P55mHyQ9eu8Z5XJO/Nm7e6gF3fvtCmTegyyjFKCjE6+ugwoFtq773hnXeSi0dEMsAdhg+HSy4JU0179w7/2XOUkkJMLrooJIQDDoC//z3s2267ZGMSkTqaNi3UKxozJtTp3nffpCOqMyWFLFqwIKyHsXRpqB4JoSx6jRbYEJH66fHHQ5VCd7jvPjjvvLyoQaKkkCXu4R6V/v2hRYtQY14JQSSPtGwZBpQHDYK2bZOOJmMsLGWQm4qKinx8aid9PfLhh2G9Vgg3MbZvn2w8IlJHK1aEuuErVsDf/hb2ldYsyjFm9pG7V7iMW+63deqR666D9dYLj9KuxUGDlBBEct6ECaGs9TXXwBdfhGQAOZkQqqPuowy5885Q1ni99eCUU8K+Zs1C/XsRyVG//hoWs7j99tAP/M9/wh/+kHRUWaWkUEf/+U8YUL7mGmjaFM4/P0xRFpE8UFwcPvGddlroOtpww6QjyjolhTqYMgX22Wf18+uuy9GFwkVktcWL4fnnQzN/hx3CHacJroQWN40p1MK118Laa0OnTuF5v37w8cdw6aXJxiUidTRmDGy/PZx+Onz5ZdhXQAkB1FKolYkTQyvyzDNhnXXgrLNg/fWTjkpEam3+/PCp7rHHwl2l77yTMwXsMk1JIQ3/+Q/MmbP6+ezZsMUWcMstycUkIhlSWsCuuDj0Af/1rzlVwC7TlBSqMXfumuMGpQ46KP5YRCSD5s4NK581bBjuNG3bFn73u6SjSpzGFKpw7bWwySZhu0+fsNZ26eO555KNTURqyR0eeSSsczBkSNjXo4cSQkQthUr8+GPoHtpkk1DM7pxz6uVyqiJSE999F6qYvvZa6ALYf/+kI6p3lBQqcd994WvXrqGbUURy3PDhoYCdGQwcGKqb5kEBu0xTUqjEzTeHr3femWwcIpIhm24a6s88+GBYAEcqpKRQibXXhmOOgdatk45ERGplxYpQnmLlyrDU4SGHhIdUSW2nSqy1Vli6UkRy0McfQ5cuYXrp5MmrC9hJtZQURCR//PILXH11qFv/44+hXMUTT+RlNdNsiS0pmFk3M5tsZsVmdnUFr7cxszfNbIKZTTSzw+OKTUTyxNSpcNddcMYZocR1Dq+VnJRYkoKZNQQGAIcBnYATzaxTucP+Cjzj7rsAJwAD44hNRHLcTz+FuvUQ6hZNmQJDhxZERdNsiKulsCtQ7O5T3X058BTQo9wxDpRWENoA+D6m2NYwbhx07x7WVRaRem706FDJtGfP1QXs8mhpzCTElRRaATNSns+M9qW6ATjFzGYCo4ELK3ojM+ttZuPNbPzcuXMzHuioUfDSS1BUBAcfnPG3F5FMmDcvlLY+4oiwmtW77xZsAbtMq08DzScCj7r7FsDhwHAz+0187j7Y3Yvcvahly5YZD2LJknA/y/vvQ7duGX97Eamr0gJ2Tz0Vppp+/DHsvnvSUeWNuO5TmAWkzvjfItqXqifQDcDd/2tmTYAWwBxiMmIE9O8f6mOJSD3z44/QsmX4D3rnnaGbaKedko4q78TVUvgQ6GBmW5pZY8JA8qhyx0wHDgQws45AEyDz/UNVmDYtfH3mmTjPKiJVcoeHHoJtt4XBg8O+7t2VELIklqTg7iXABcAY4EvCLKNJZtbHzI6KDrsM6GVmnwIjgDPck7nj5HBNhhWpH6ZODXXqzzorVDFVzfqsi63MhbuPJgwgp+67PmX7C2CvuOKpyMqVSZ5dRNYwbBicd17oLnrwQejVSwXsYqDaRylKW6b6vROpBzbfHA44AB54ICx1KLFQUoiUlMD06bDBBtC4cdLRiBSg5cvh1lth1Sq44YYwJ1zzwmOXdlIws4MJA8SbuHt3MysC1nf3N7IWXUxefRXefjts77tvsrGIFKQPP4Q//xk+/zzcf+CuekUJSaujxMwuBB4ApgClfzZ/AW7KUlyxOvRQ6Ns3bJ99drKxiBSUpUvh8svDfQYLFoS7Rx97TAkhQen2nl8CHOTutwKron1fAdtmJaoYffBB+PrnP4ffySOOSDYekYLy7bdhmcNevWDSpDDVVBKVbvdRM1aXqSidJroWsDzjEcXshx/C1z/9CZo3TzYWkYKwaBE89xyceWYoYFdcrNWs6pF0WwpjgfLlri8C3sxsOPH75pvwtUWLZOMQKQgvvxwSwVlnwVdfhX1KCPVKuknhQuAYM/sOaGZmk4HjgUuzFVhcSruPNtkk2ThE8trcuXDyyXDkkaGk9X//C9ttl3RUUoG0uo/cfbaZdQG6AG0JXUkfuPuqqr+z/mvSBP7v/zQNWiRrVq6EvfcO4wc33hhWRtO873orraRgZi+6ew/gg+hRuv85d/9DtoKLi34/RbLghx9CE7xhQ+jXD9q1C2sfSL2WbvfR/pXs75qhOEQkX6xaBYMGwTbbhK8Quo2UEHJClS0FM+sTbTZO2S61FTAtK1HFaPFi1TwSyZji4jC99K23QomKQw9NOiKpoeq6j0qnBTRgzfUQnDCucEMWYorVyJGw8cZJRyGSBx55JBSwa9wYhgwJS2TqJrScU2VScPczAczsPXcfEk9I8WraFHbeOekoRPJAmzahZTBgALQqv9qu5Ip0Zx8NATCzZoTV0CzltanZCS0ea60FO+6YdBQiOWjZMrjlljCG0KcPHHhgeEhOS3f2UUfgSWBnQteRsfrO5pxdvPLnn8PNlSJSQ++/H7qHJk2C009XAbs8ku7sowcIdy9vBPwEbAgMAk7PUlyx2G+/8LVJk2TjEMkZS5bApZfCHnuET1QvvQSPPqqEkEfSrX20M3Cwu68wM3P3RWZ2BfA58Hj2wsueV1+FCRNCvaMrrkg6GpEcMW0aDBwI55wT1j5Yf/2kI5IMS7el8CuhAB7APDNrE31vzs7beeed8PWf/9TsI5EqLVwIQ4eG7U6dwrTTgQOVEPJUuknhHUKtI4CRwL+Bt4GcXmCnQYMwlVpEKvHiiyERnHPO6gJ2qgmT19KdfXR8ytNrgUnAesCwbAQlIgmbMwcuugiefhp22iksfqMCdgWhxms0R0XwhptZY6AXMCDjUYlIclauhL32CouW33QTXHllmLstBaHapGBmBwK/A4rd/UUzawScB1wF/A8lBZH88P33oWRww4Zw772hgF2nTklHJTGrckzBzK4CXiSMJzxhZrcD7wHHAb3dXbd9ieS6VavggQdC99CDD4Z9hx+uhFCgqmspnA3s5+4fmdnuwLvAZe5+T/ZDE5Gs+/rrUMBu7Fg46CA47LCkI5KEVTf7qIW7fwTg7uOAZcC9WY8qBsOHhw9IIgXroYdC4a+JE+Hhh8PNO1tumXRUkrB0xhSMUNbCCPcrYGZlySRXV19bd93QdSpSsNq1Cy2DAQNgs82SjkbqieqSwnpAScpzS3leWv8oJ/+0msHRRycdhUiMli2Df/wjbN90kwrYSYWqSwpqS4rkg/feCwXsvvoK/vxnFbCTSlW3nkLOr6wmUtAWL4brroP77oPWreGVV7QamlQp3TIXdWZm3cxsspkVm9nVlRxzvJl9YWaTzOzJuGITyVvTp4d1ks8/Hz7/XAlBqlXjO5prw8waEm5yOxiYCXxoZqPc/YuUYzoA1wB7ufsCM9skjthE8s6CBfDss9C7d7jXYOpU2HzzpKOSHBFXS2FXwh3RU919OfAU0KPcMb2AAe6+AMDd58QUm0j+eP75kAjOOw8mTw77lBCkBmqUFMysdXQTW021AmakPJ8Z7Uu1DbCNmb1rZuPMrFslMfQ2s/FmNn7u3Lm1CEUkD/3wAxx3HPzhD6FUxQcfwLbbJh2V5KC0koKZtTGzd4GvgP8X7fujmQ3NYCyNgA5AV+BEYIiZNS9/kLsPdvcidy9q2bJlBk8vkqNWroR99oF//Qv69g0JoXPnpKOSHJXumMIg4GVgH2B+tO81oF+a3z8LaJ3yfItoX6qZwPvuvgL41sy+JiSJD9M8h0hhmTkzdA01bAj9+4e7kVXeWuoo3e6jXYFbo7uXHcDdFwEbpPn9HwIdzGzLqOT2CcCocse8QGglYGYtCN1JU9N8f5HCsWpVmGK63XahkB2EO5OVECQD0k0KPwLtU3eYWSdgejrf7O4lwAXAGOBL4Bl3n2RmfczsqOiwMcB8M/sCeBO4wt3nV/yOIgXqq69g333DAjh77w1HHpl0RJJn0u0+uhN4ycxuARqZ2YmEFdhuTfdE7j4aGF1u3/Up2w5cGj1EpLyhQ+GCC6BpUxg2DE49VXclS8aluxznw2Y2n1BKewZwGvA3d38hm8GJSIqtt4bu3eH++2HTTZOORvJUWknBzBq6+4uEBXdEJA6//gp9+oTtvn1h//3DQySL0h1T+MHMBprZXlmNRkSCd9+F3/0ObrkF5s4NBexEYpBuUjgEWAyMMLNvzewWM9NSnCKZ9vPPcOGF4b6DZctgzBgYMkRjBxKbtJKCu09w9yvdvQ1wBrAh8IaZTcxmcCIFZ+bMMKB84YXw2WdwyCFJRyQFpjYF8b4iTCudTri5TETqYv58eOYZOPdc6NgxFLDTSmiSkHTLXDQ3s55m9jrhhrKuwG2AKpmK1JY7jBwZCthddNHqAnZKCJKgdFsK3wPvAU8Cx7r7wuyFJFIAZs8Oaxw8/zz8/vfw6qsqYCf1QrpJYWt3n53VSEQKRWkBu1mz4Pbb4S9/gUaxLG0iUq1KfxPNbF93Hxs97WhmHSs6zt3fyEpkIvlmxgxo1SoUsBswIBSw22abpKMSWUNVH08GAjtE2w9VcowDW2U0IpF8s3JlSALXXBNaBuefr2Uxpd6qNCm4+w4p21vGE45InvnyS+jZE/7731DJtHv3pCMSqVK6s48qLG9hZs9lNhyRPDJ4cLgr+euvYfhwePllaNMm6ahEqpTu6FZlBVe6ZigOkfzToQMcc0xYAGcTzd6W3FBlUjCzqBoXjVO2S20FTMtKVCK56Jdf4IYbQkmKW29VATvJSdV1H7WOHg1StlsTltOcARyX1ehEcsXYsbDzzmEgedEiFbCTnFVlS8HdzwQws/fcfUg8IYnkkJ9+gquvDstibrUVvP46HHBA0lGJ1FpV9ym0c/fvoqevm1mFU0/dXesoS+H6/nt49FG49NKw9sG66yYdkUidVNVS+AxoFm0XE+5JKF+/14GGWYhLpP6aNy8UsDvvPNhuO/j2W62EJnmj0jEFd2+Wst3A3RtGX1MfSghSONzh6adDAbtLLglTTUEJQfJKuovsrMHMtjKzdpkNRaQe+/57OPpoOOEEaNsWPvpIJSokL6V789oIM9sz2j4TmARMMrOe2QxOpF5YuRL23TdUMr3zznB38o5aeFDyU7o3rx0InB5tXwocBCwEXqDyukgiuW3aNNhii1DAbuDAMLuoffukoxLJqnS7jxq7+3IzawVs5O7vuvskQJ2pkn9WroS77gqroD3wQNh3yCFKCFIQ0m0pfGJm1wBtgZcBogTxU7YCE0nE55+HAnYffABHHhnGEUQKSLothZ7AjsA6wN+ifXsAT2QjKJFEPPggdO4c1kh+8kkYNSp0H4kUkLRaCu7+DXBSuX0jgZHZCEokVu6hXlHHjnDccXDPPdCyZdJRiSQi7TUAo1lHpwKtgFnAcHd/JFuBiWTd0qVw/fVhIPm222C//cJDpIClOyX1OuBq4CngoujrldF+kdzz1luw007Qrx8sXqwCdiKRdFsKZwFd3b2sVLaZjQHGAjdnIzCRrFi0CK68MiyAs/XW8MYbKm8tkiLdgeZ1gbnl9s0nDDyL5I7Zs+Hxx+Hyy2HiRCUEkXLSTQqvAE+Y2bZmto6ZbQcMA8akeyIz62Zmk82s2MyuruK4Y83Mzawo3fcWqdLcuXDffWF7u+3gu+/gjjugadNEwxKpj9JNChcAPwMTgcXAJ8AS4MJ0vtnMGgIDgMOATsCJZtapguOaARcD76cZV624w6RJ2TyD1AvuYWppx45w2WWrC9hpZpFIpapNCma2AdAeOJ/QXbQZ0NTdT3P3hWmeZ1eg2N2nuvtywkB1jwqO+wdwG/Brmu9bK9Onh68L041ecs+MGdC9O5x8crgTecIEFbATSUOVScHMjgC+B8YDM4H93H2Ou6+q4XlaEZbvLDUz2pd6rs5Aa3d/uZqYepvZeDMbP3du+WGO9KxYEb6efnrVx0mOKimBrl3hzTfh7rvh3Xdh++2TjkokJ1TXUvgHcBWwHnA9WZppZGYNgLuAy6o71t0Hu3uRuxe1rGM3gJVfMkhy23ffhbpFjRrBoEHw2Wdh3YOGWvZDJF3VJYWt3P1+d19KGBOobUWwWUDrlOdbRPtKNQN2AN4ys++A3YFRGmyWtJSUhJLWHTuGaqYABx0UqpqKSI1Ud59CWdJw9xIzS/sO6HI+BDqY2ZaEZHACKWUz3H0R0KL0uZm9BVzu7uNreT4pFBMnhgJ248dDjx5w7LFJRySS06r7I9/UzMamPG9W7jnuvm91J4kSygWEKawNgYfdfZKZ9QHGu/uomgYuwsCBcPHFsOGGYZnM445Tn6BIHVWXFMqvrFbrBXXcfTQwuty+6ys5tmttzyMFoLSA3Q47hOUx774bWrSo/vtEpFpVJgV3HxZXICLVWrIE/vrXMJB8xx1hicx9q22oikgNpHvzmkiyXn89rIt8zz2wbJkK2IlkiZKC1G8LF8JZZ4XZRI0awdix0L+/xg5EskRJQeq3H3+Ep56Cq66CTz+FffZJOiKRvFbbKaYi2VOaCC6+GLbdNtyUpoFkkViku8jO2mZ2s5lNNbNF0b5DommmIpnhHspad+oU1jyYMiXsV0IQiU263Ud3E+44PhkoHeGbBJybjaCkAE2fDkccAaeeGloHn3wCHTokHZVIwUm3++gYoL27LzGzVQDuPsvMWlXzfSLVKy1gN2dOGEQ+7zzVKxJJSLpJYXn5Y82sJWH1NZHamToV2rYNs4qGDAnLY7Zrl3RUIgUt3e6jZ4FhUe0izGwz4H7CuggiNVNSArfdFsYOBgwI+w48UAlBpB5INylcC3wLfAY0B6YQ1lm4MUtxSb765BPYbTe4+mo4/PBQr0hE6o20koK7L3f3v7j7esCmQLPo+fLshid55f77oUsXmDULRo6E556DzTZLOioRSZHWmIKZlS9M38yiO0rdfWqmg5I8U1rAbqedwvKYd90FG22UdFQiUoF0B5qLCVNRU2sLlE5N1TQRqdjixXDddbDWWmERHBWwE6n30u0+auDuDaOvDYDNgcHAqVmNTnLXq6+G0tb33RcWxVYBO5GcUKvaR+7+A3AJcEtmw5Gct2ABnHkmHHooNGkSCtjde68K2InkiLoUxNsWaJqpQCRPzJkTBpGvuSbMNNp776QjEpEaSHeg+R1WjyFASAbbA32yEZTkmB9+gBEj4C9/WV3AbuONk45KRGoh3YHmoeWeLwE+dfcpGY5Hcok7PPZYSAZLl8KRR4Z6RUoIIjmr2qRgZg2BA4De7r4s+yFJTvjuOzj77DCgvNdeMHSoCtiJ5IFqk4K7rzSzQ4BVMcQjuaCkBPbfH+bNC2UqzjkHGmi9JpF8UJPS2Tea2VrZDEbqueJiWLkyFLB7+GH4/PNQ0VQJQSRvVPm/2cxOjDYvBK4AfjazGWY2vfSR9QgleStWQN++sP32qwvY7b9/qHAqInmluu6jQcAI4JQYYonN2LHhq+6nSsPHH0PPnmF66XHHwZ/+lHREIpJF1SUFA3D3t2OIJTbzo1UgtAZ8Nfr3h0svhZYtQ/G6Y45JOiIRybLqkkJDM9ufNWsercHd38hsSPFp2TLpCOqp0gJ2u+wCp50G/frBhhsmHZWIxKC6pLA28BCVJwUHyldQlVz188/hTuS11w6JYJ991JwSKTDVJYUl7q4/+oXglVfCfQczZsAll6xuLYhIQdFcwkI3fz6cfjocdhisuy68+25Y70AJQaQgVZcU9Jch382fD88/D3/7G0yYAHvskXREIpKgKpOCuzfL1InMrJuZTTazYjO7uoLXLzWzL8xsopm9bmaaBJ8ts2eHRW/cYZttYNo06NMnjCWISEGLpfsoqp80ADgM6AScaGadyh02AShy952AkcDtccRWUNzDncgdO4aWQXFx2K+ZRSISiWtMYVeg2N2nuvty4CmgR+oB7v6muy+Nno4DtogptsLw7bdwyCHhRrSdd4ZPP1UBOxH5jXRLZ9dVK2BGyvOZwG5VHN8T+HdFL5hZb6A3QJs2bTIVX34rKYEDDgjjBw88AL17q16RiFQorqSQNjM7BSgC9qvodXcfTFgfmqKiIhWqqMqUKbDVVqGA3SOPwNZbQ+vWSUclIvVYXB8XZwGpf422iPatwcwOAq4DjtLaDXWwYgXcdBPssAPcf3/Y17WrEoKIVCuulsKHQAcz25KQDE4ATko9wMx2IRTg6+buc2KKK/+MHx/GDSZOhBNOgBNPrP57REQisbQU3L0EuAAYA3wJPOPuk8ysj5kdFR12B7Ae8KyZfWJmo+KILa/cey/stltY/ObFF8O6yZtsknRUIpJDYhtTcPfRwOhy+65P2T4orljyTmlJiqKi0Eq4/XZo3jzpqEQkB9W7gWapgZ9+gquugiZN4O67w1rJe+2VdFQiksM0LzFXjR4dVkIbPDjMLtKKQSKSAUoKuWbePDjlFDjiCNhgA3jvPbjjDhWwE5GMUFLINQsWwL/+BX//e1gqc7eq7gEUEakZjSnkglmz4Ikn4IorQmmKadM0kCwiWaGWQn3mDkOGQKdOcMMN8M03Yb8SgohkiZJCffXNN3DggaFOUefO4Wa09u2TjkpE8py6j+qjkpKQEP73Pxg0CM46SwXsRCQWSgr1yeTJoWhdo0YwbFjY3kIVxEUkPvr4WR8sXw433gg77ggDBoR9++2nhCAisVNLIWkffBBKU3z+OZx0Epx8ctIRiUgBU0shSffcA3vssfregyeegBYtko5KRAqYkkISSktS7Lor9OoFkybBkUcmG5OICOo+iteiRXDllbDOOqGVsOee4SEiUk+opRCXf/0r3IQ2dCisvbYK2IlIvaSkkG1z54YB5KOOgo03hnHj4LbbVMBOROolJYVsW7QolLm+8cawVGaXLklHJCJSKY0pZMOMGfD443D11aE0xbRpocy1iEg9p5ZCJq1aBQ8+GBa/uemm1QXslBBEJEcoKWTKlClwwAFw7rlhqulnn6mAnYjkHHUfZUJJCRx8MCxcCA89BGeeqYFkEclJSgp18eWXYdGbRo1g+PBQwG7zzZOOSkSk1tR9VBvLloXlMHfaCe6/P+zbZx8lBBHJeWop1NS4caGA3RdfwKmnhoeISJ5QS6Em+vULZSl+/jnce/DYY+GGNBGRPKGkkI5Vq8LXPfaAc84JZThA3IoAAArFSURBVK4POyzZmEREskDdR1VZuBAuuwyaNoX77lMBOxHJe2opVOaFF0IBu2HDoFkzFbATkYKgpFDenDlw/PFwzDGw6aZhZbS+fXXfgYgUBCWF8n76CV57DW6+OSSEzp2TjkhEJDYaUwCYPj3cfHbttaE0xfTpoctIRKTAxNZSMLNuZjbZzIrN7OoKXl/bzJ6OXn/fzNplPahVq2DgwFDArm/f1QXslBBEpEDFkhTMrCEwADgM6AScaGadyh3WE1jg7u2Bu4HbshnTNkymSbeucP75YarppEkqYCciBS+ulsKuQLG7T3X35cBTQI9yx/QAhkXbI4EDzbIzumsrSxjDoTT44jN45BEYMwbatcvGqUREckpcSaEVMCPl+cxoX4XHuHsJsAj4ze3CZtbbzMab2fi5c+fWKpgOHRvxUNfHWfbxF3DGGZpZJCISybmBZncfDAwGKCoqqtXNAz16QI8ee2c0LhGRfBBXS2EW0Drl+RbRvgqPMbNGwAbA/FiiExERIL6k8CHQwcy2NLPGwAnAqHLHjAJOj7b/CLzhrtuIRUTiFEv3kbuXmNkFwBigIfCwu08ysz7AeHcfBTwEDDezYuB/hMQhIiIxim1Mwd1HA6PL7bs+ZftX4Li44hERkd9SmQsRESmjpCAiImWUFEREpIySgoiIlLFcnvVpZnOBabX89hbAvAyGkwt0zYVB11wY6nLNbd29ZUUv5HRSqAszG+/uRUnHESddc2HQNReGbF2zuo9ERKSMkoKIiJQp5KQwOOkAEqBrLgy65sKQlWsu2DEFERH5rUJuKYiISDlKCiIiUibvk4KZdTOzyWZWbGZXV/D62mb2dPT6+2bWLv4oMyuNa77UzL4ws4lm9rqZtU0izkyq7ppTjjvWzNzMcn76YjrXbGbHR//Wk8zsybhjzLQ0frfbmNmbZjYh+v0+PIk4M8XMHjazOWb2eSWvm5n1j34eE82sc51P6u55+yCU6f4G2ApoDHwKdCp3zHnAg9H2CcDTSccdwzXvDzSNts8thGuOjmsGjAXGAUVJxx3Dv3MHYAKwYfR8k6TjjuGaBwPnRtudgO+SjruO17wv0Bn4vJLXDwf+DRiwO/B+Xc+Z7y2FXYFid5/q7suBp4Ae5Y7pAQyLtkcCB5rl9KLN1V6zu7/p7kujp+MIK+HlsnT+nQH+AdwG/BpncFmSzjX3Aga4+wIAd58Tc4yZls41O7B+tL0B8H2M8WWcu48lrC9TmR7AYx6MA5qb2WZ1OWe+J4VWwIyU5zOjfRUe4+4lwCJg41iiy450rjlVT8InjVxW7TVHzerW7v5ynIFlUTr/ztsA25jZu2Y2zsy6xRZddqRzzTcAp5jZTML6LRfGE1piavr/vVqxLbIj9Y+ZnQIUAfslHUs2mVkD4C7gjIRDiVsjQhdSV0JrcKyZ7ejuCxONKrtOBB51935mtgdhNccd3H1V0oHlinxvKcwCWqc83yLaV+ExZtaI0OScH0t02ZHONWNmBwHXAUe5+7KYYsuW6q65GbAD8JaZfUfoex2V44PN6fw7zwRGufsKd/8W+JqQJHJVOtfcE3gGwN3/CzQhFI7LV2n9f6+JfE8KHwIdzGxLM2tMGEgeVe6YUcDp0fYfgTc8GsHJUdVes5ntAgwiJIRc72eGaq7Z3Re5ewt3b+fu7QjjKEe5+/hkws2IdH63XyC0EjCzFoTupKlxBplh6VzzdOBAADPrSEgKc2ONMl6jgNOiWUi7A4vcfXZd3jCvu4/cvcTMLgDGEGYuPOzuk8ysDzDe3UcBDxGamMWEAZ0Tkou47tK85juA9YBnozH16e5+VGJB11Ga15xX0rzmMcAhZvYFsBK4wt1zthWc5jVfBgwxs78QBp3PyOUPeWY2gpDYW0TjJH8H1gJw9wcJ4yaHA8XAUuDMOp8zh39eIiKSYfnefSQiIjWgpCAiImWUFEREpIySgoiIlFFSEBGRMkoKklPM7C0zOyvpOKpiZieb2atVvL6PmU2OMyaRdCkpSGLM7Dsz+8XMFqc8Nk8gjrfM7Nfo/PPM7Lm6FBVz9yfc/ZCU93cza5/y+jvuvm1d4y7PzG4wsxXRdSw0s/eiUg/pfv8acUphUlKQpHV39/VSHklVtbzA3dcj3PXbHLg7oTjq6unoOloAbwLPJhyP5BglBalXzGxDM3vJzOaa2YJou8LS3mbW3szeNrNF0Sf8p1Ne287MXjOz/0WLshyfzvnd/X/APwm1kjCzPc3sw+gcH5rZninnOMPMpprZz2b2rZmdnLL/P9H22OjwT6NP8H8ys67R3amY2VVmNrLcdd1rZv2j7Q3M7CEzm21ms8zsJjNrmMZ1lABPAK3MrGX0Xrua2X+jVsRsM7s/KhdRYZzR/iPN7JOUlsdO6fwcJXcpKUh90wB4BGgLtAF+Ae6v5Nh/AK8CGxIKgd0HYGbrAq8BTwKbEEqXDDSzTtWdPKoRdCwwwcw2Al4G+hPKqd8FvGxmG0fn6A8c5u7NgD2BT8q/n7vvG23uHLWEni53yFPA4WbWLDp/Q+D4KHaAR4ESoD2wC3AIUO2YSvTH/jRCcccF0e6VwF8IrYg9CDWCzqssTgs1sh4Gzo6ufxChkODa1Z1fcpeSgiTthehT6EIze8Hd57v7P919qbv/DNxM5aW9VxCSx+bu/qu7/yfafyRhxa1H3L3E3ScQPv0fV0Uc/c1sIWE1r9nApcARwBR3Hx69zwjgK6B79D2rgB3MbB13n+3uk2p68e4+DfgYOCbadQCw1N3HmdmmhLo2l7j7kqh44d1UXZ/r+Og6fiEssvPHqNWAu3/k7uOia/mO8Ee+qrLpvYFB7v6+u69092HAMkKVWclTSgqStKPdvXn0ONrMmprZIDObZmY/EZbPbF5Jl8mVhGUIP7CwBvGfo/1tgd1Sks1C4GTg/6qI46IohlbufrK7zwU2B6aVO24a0MrdlwB/As4BZpvZy2a2XS1/Bk8S1gEAOInVrYS2hOJns1OuYxCh9VOZZ9y9ObAp8Dnw+9IXzGybqDvuh+hn25eqy0q3BS4r93NsTfi5SJ5SUpD65jJgW2A3d1+fsEYthD/+a3D3H9y9l7tvTujiGBjNnpkBvJ2SbJpHXSLn1jCW7wl/GFO1IapX7+5j3P1gYDNCC2JIDd+/1LNA12js5BhWJ4UZhE/mLVKuY3133766N3T3eYRP+jekzKR6IIqzQ/SzvZYKfq4pZgA3l/s5No1aTJKnlBSkvmlG6PpYGPXp/72yA83suJRB6AWEUsmrgJcIy1CeamZrRY8uFurr18To6H1OMrNG0eBrJ+AlM9vUzHpEYwvLgMXRuSvyI2Gx+QpFrZK3CGMp37r7l9H+2YQxk35mtr6ZNTCzrc0srZXy3H0yocz0ldGuZsBPwOKoVVM+SZaPcwhwjpntZsG6ZnZE6fiH5CclBalv7gHWAeYRFsN5pYpjuwDvm9liwmIjF3tY1P1nwoDsCYRP+z8AtwE1GiCN1h44ktB6mU/443pk9Cm8AWHc4XvCOhz78ds/sqVuAIZFXTCVzYJ6EjiI1a2EUqcBjYEvCIlvJKFlkq47gN5mtglwOaF76mfCH/zyg95rxBktQtSLMNC/gFCz/4wanFtykNZTEBGRMmopiIhIGSUFEREpo6QgIiJllBRERKSMkoKIiJRRUhARkTJKCiIiUkZJQUREyvx/ZDL7+IckIA0AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_i2h36hKjR3o",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08cc4035-e437-4c72-c4ee-73237e5dc4f5"
      },
      "source": [
        "metrics.auc(fpr, tpr)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9458050217460783"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zfprjR02bspM"
      },
      "source": [
        "We get a higher accuracy using a Discriminator model over `LabelModel`. This is in part because the discriminative model generalizes beyond the noisy labeling function labels and makes good predictions on all data points, not just the ones covered by the labeling functions. \n",
        "\n",
        "Now, we look at the precision and recall metrics to better understand the model performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IVZMM5B0LKFK"
      },
      "source": [
        "# storing predicted binary values by the model\n",
        "pred = sklearn_model.predict(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2xqDUVjEMvIF",
        "outputId": "3d61806c-d9a4-490d-ca3b-120a8a570331"
      },
      "source": [
        "# check classiification metrics\n",
        "print(classification_report(Y_test, pred))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.95      1.00      0.97       966\n",
            "           1       0.97      0.64      0.77       149\n",
            "\n",
            "    accuracy                           0.95      1115\n",
            "   macro avg       0.96      0.82      0.87      1115\n",
            "weighted avg       0.95      0.95      0.94      1115\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 211
        },
        "id": "BRfnnJMoOafB",
        "outputId": "083f83ea-58f1-4250-abb5-a4438412ac1f"
      },
      "source": [
        "# plot the confusion matrix\n",
        "cm = confusion_matrix(Y_test, pred)\n",
        "\n",
        "plt.rcParams['xtick.top'] = plt.rcParams['xtick.labeltop'] = True\n",
        "plt.rcParams['xtick.bottom'] = plt.rcParams['xtick.labelbottom'] = False\n",
        "fig, ax = plt.subplots(figsize=(4, 3))\n",
        "ax.imshow(cm)\n",
        "ax.grid(False)\n",
        "ax.xaxis.set(ticks=(0, 1), ticklabels=('Predicted 0s', 'Predicted 1s'))\n",
        "ax.yaxis.set(ticks=(0, 1), ticklabels=('Actual 0s', 'Actual 1s'))\n",
        "ax.set_ylim(1.5, -0.5)\n",
        "\n",
        "for i in range(2):\n",
        "    for j in range(2):\n",
        "        ax.text(j, i, cm[i, j], ha='center', va='center', color='red')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOYAAADCCAYAAABdV76bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPYElEQVR4nO3deZBV5Z3G8e/TG0hDNzQ0UWwU48REHRIG2pGJu9EEsRyXQQNYo6Y0oBlhNA4mNclYTFWclKMzqSgTRyQVNdE4o7hjQIkorrggi8E1biCC7Euz9fKbP+5p7WmgN9D7dvN8qm71ue95zzm/+/Z97nnPuU2hiMDM0lKQ7wLMbGcOplmCHEyzBDmYZglyMM0S5GCaJcjBbIGkekkLJL0m6R5JPfZgX7dJGpUtT5N0RAt9T5T0zQ4c431J/XbRPkzSYknvSLpRktq77z3RhcbxWklLJW1u7z7by8Fs2daIGBIRfwnsAC5tulJSUUd2GhGXRMSSFrqcCLT7DdWCm4HvA1/JHiP24r7boquM48PAX+/F/e2Wg9l2TwN/kX0KPy3pIWCJpEJJ10t6SdIiSeMBlDNF0puSZgP9G3ck6UlJ1dnyCEnzJS2U9EdJg8i9ca/MzjLHSaqUND07xkuSjsm27SvpMUl/kjQN2OlMKOkAoCwiXojcX5PcAZyVrZsoaUlW992f49g11SnHESAbw4+bt0s6N5sNLJQ0d6+MUkT4sZsHsDn7WQQ8CFxG7lO4BjgkWzcO+Gm23A14GTgEOAd4HCgEBgDrgVFZvyeBaqASWNpkXxXZz8nAPzWp4y7g2Gz5IOD1bPlG4Jps+XQggH7NXkM1MLvJ8+OAR7Ll5UC3bLm3x3H347ir19Pk+WLgwL05jh2aQuxD9pO0IFt+Gvg1uanRixHxXtb+beDrjdc9QDm56eLxwO8joh5YLumJXex/ODC3cV8RsXY3dZwCHNHk0rBMUs/sGOdk286QtK6dr28RcKekB4AH2rlte3T1cXwWuE3S/wL3tXPbXXIwW7Y1IoY0bch+qTVNm4AJETGrWb+Re7GOAmB4RGzbRS2t+QioavK8KmuD3NnheOAM4CeSBkdE3Z6Xu5OuMI67FRGXSjqa3Hi+ImlYRKzZk336GnPPzQIuk1QMIOkwSaXAXOC72bXTAcBJu9j2BeB4SYdk21Zk7ZuAXk36PQZMaHwiqfFNPhcYm7WdBvRpfoDIXRNtlDQ8uxt7AfCgpAJgYETMAX5E7gzVsyMDsJckPY4tkXRoRMyLiGuAVcDA9my/Kw7mnpsGLAHmS3oNuIXcTOR+4O1s3R3A8803jIhV5K6t7pO0EPifbNXDwNmNNy2AiUB1dlNkCZ/d1fxXcm/IP5Gbin24mxp/kNX5DvBn4A/krtl+J2kx8CpwY0Ss7/gw7LHkx1HSv0taBvSQtEzS5GzV9cp9HfUa8BywcE8GAkDZBauZJcRnTLMEOZhmCXIwzRLkYJolyME0S5CDmSBJ4/JdQ1fQmcfRwUxTp31DJabTjqODaZagfe4PDPpVFMaggcX5LqNFq9bUU9m3MN9ltOitRR3+t85fmFq2U0y3fJfRok2sWx0Rlc3b97k/Yh80sJgXZ+3xnzLu874zYEjrnaxVs+PeD3bV7qmsWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLME7XP/1XsSbl2P7twIAXF+GYzrnWv/9Xr0mw1QKDilB/Ev/eDVbWjSJ7n1AXFVBYzsmb/aE1cc9fwnT1JMA4UET3Mgd+jIfJfVbm0KpqSzgPuBwyPijVb6XgFMjYgtHSlI0kVAdURc3qxdwC+BkcAW4KKImN+RY+TVG9vRnRuJR6ugRGjscuLUHrC8Ds2qIf54EHQTrK7L9f9qCTFzIBQJVtahby0lvl2ae247qaWASZzANhVRGA38gjm8FPvzuvrmu7R2aetUdgzwTPazNVcAPTpc0e6dBnwle4wDbv4cjvH5e7sWhnaDHgVQJGL4fvBoDbp9I3F5n1woAfpln5lZPwC2BziPLZPYptzYFdFAEUHkuaSOaDWYknoCxwIXA6ObtBdKukHSa5IWSZogaSIwAJgjaU7Wb3OTbUZJui1bPkPSPEmvSpot6UutlHImcEfkvAD0lnRA9pgraUFWy3HtHIMv1ldLYN42WFsPWxrQEzVoeR28uwPN24pGLkVnL4MF2z7bZv42dMKH6KQPiesqfbZsRUEE/x2Pcw8PM5/+vNHJzpbQtqnsmcDMiHhL0hpJwyLiFXJnrUHAkIiok1QREWsl/RA4KSJWt7LfZ4DhERGSLgGuBq5qof+BwNImz5dlbScAsyLiWkmF7OJsLWlcVi8HHZjny+rDSoh/6INGL4cegiO75T4e64D1DcSMKliwHY1bQcw7GCQY2p146iB4awf6x5XEyT2gu+/b7U6DxKWcSmnsYDLPMyg28L7K811Wu7TltzsGuDtbvpvPprOnALdERB1ARKxt57GrgFmSFgOTgI5eob8EfE/SZGBwRGxq3iEipkZEdURUV/Yt7OBh9qKxZcRjA4kHqqC8kDi0BA4oIkaW5oL4V91zv5k1Df9/u8NKoLQA3tiRl7I7mxqVsJBKqlmR71LarcVgSqoATgamSXqfXIDOy27EtFXTKX73Jss3AVMiYjAwvtm6XfkIGNjkeRXwUUTMBY7P1t8m6YJ21JYfjTd2ltXCo5vh7J7EiFL07NZc+593QC3QtwA+rIW6bAiX1sI7O2BgcV7K7gzKYzulkfvgKol6hrKSpfTKc1Xt19q8bhTw24gY39gg6SngOOBxYLykOU2nssAmoBfQOJVdKelw4E3g7Gw9QDm5MAFc2IZaHwIul3Q3cDSwISI+lnQwsCwibpXUDRgK3NGG/eWNLl4B6+qhWMTPK6G8EMaUwZUr0Ykf5tp/2T939py3FU1ZD8WAsv4pnPUTVcFWruZlCiIQwVyqmKcB+S6r3VoL5hjgumZt07P2CcBhwCJJtcCtwBRgKjBT0vKIOAn4MfAIsAp4GWj8Em4ycI+kdcATwCGt1PIoua9K3iH3dcn3svYTgUlZDZuB5M+Y8WDVzo0lIv5r/53bzy0jzi37/IvqIt5Tby7jlHyXsccU0RlvJndc9Te6x4uzBrbe0Vr0nQFD8l1ClzA77n0lIqqbt/vWnlmCHEyzBDmYZglyMM0S5GCaJcjBNEuQg2mWIAfTLEEOplmCHEyzBDmYZglyMM0S5GCaJcjBNEuQg2mWIAfTLEEOplmCHEyzBDmYZglyMM0S5GCaJcjBNEuQg2mWIAfTLEEOplmCHEyzBDmYZglyMM0S5GCaJcjBNEuQg2mWIAfTLEEOplmCHEyzBDmYZgkqyncBX7S3l/Ri5OCT811Gp1d0SK98l9A1vLvrZp8xzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZagonwXYHD72rvYomIaKKBeYmLvcz5dd86WRYzb8gLnVVzAxoLueawybWdumM+IjYsQMLNsMA+UD+P8tc8xYtNiNhTuB8DtFcfyUo8v57fQNmpTMCWdBdwPHB4Rb7TS9wpgakRs6UhBki4CqiPi8mbtXwN+AwwFfhIRN3Rk/6n6UfkZOwWvX/1mhtUuY2VBzzxV1TkcvGM1IzYu4ooDz6dWhfxsxXTmZQF8oHwo03sflecK26+tU9kxwDPZz9ZcAfTocEW7txaYCHSpQLZkfM3zTCs9Ot9lJG/gjjW82f0AthcU06ACFnev4piat/Nd1h5pNZiSegLHAhcDo5u0F0q6QdJrkhZJmiBpIjAAmCNpTtZvc5NtRkm6LVs+Q9I8Sa9Kmi3pSy3VERGfRMRLQG2z+kolzZC0MKvlu21+9YkIxL9tmMFN6+7jtG2vAzB8+/usKSjlvaK+ea4ufR+U9OPIbR/Rq34r3RpqOWrLe1TWbQLgjI0L+NWy27nyk5n0rN+W50rbri1T2TOBmRHxlqQ1koZFxCvAOGAQMCQi6iRVRMRaST8EToqI1a3s9xlgeESEpEuAq4GrOvAaRgDLI+J0AEnlzTtIGpfVS/cEp4VXlf8tawpLKW/Yys83zGBpYW9Gb32Vfy47Pd+ldQpLS/pyT/lRXPvxvWwrKObdkv40UMCMsm/w+z7DCcQF657l+2ue5Bf9R+S73DZpy1R2DHB3tnw3n01nTwFuiYg6gIhY285jVwGzJC0GJgFHtnP7RouBUyVdJ+m4iNjQvENETI2I6oioLknwBsqawlIANhTsx3Mlg/h67cfsX7+Jm9ffy+1r76JfQw1T1k+nT0OHLtv3CY+VDWZi1d9z9YDRbCrsxrLiPqwvKqVBBYTEH3oN5rDtK/JdZpu1GExJFcDJwDRJ75ML0HmS1I5jRJPlpqm4CZgSEYOB8c3WtX3nEW+RuyG0GPiZpGs6sp986Ra17New49PlobUf8WZRJaP7XsCFFWO5sGIsqwtKubz337Gu4PO4dO8ayutzH1qVdRs5puZtnuz5NfrUfXoVxTdr3uGDkn75Kq/dWpvKjgJ+GxHjGxskPQUcBzwOjJc0p+lUFtgE9AIap7IrJR0OvAmcna0HKAc+ypYv7OgLkDQAWBsRv5O0Hriko/vKhz4NW7lm42MAFBLM6XYor5QMzHNVnc9PVz5EWf1W6lTIr/p9i5rC7lz2yaN8efsqEKwsKuPGfqfmu8w2ay2YY4DrmrVNz9onAIcBiyTVArcCU4CpwExJyyPiJODHwCPAKuBloPEibzJwj6R1wBPAIS0VImn/bPsyoCH7WuYIYDBwvaQGcjeGLmvlNSVlRWEZP+gzqsU+F1aM/YKq6bwmDRi9U9sN/UfmoZK9QxHReq8upLy4Mv6myRf41jEq65XvErqEme/+xysRUd283X+SZ5YgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIcTLMEOZhmCXIwzRLkYJolyME0S5CDaZYgB9MsQQ6mWYIUEfmu4QslaRXwQb7rMMscHBGVzRv3uWCadQaeypolyME0S5CDaZYgB9MsQQ6mWYL+Dx6lScDLe3+UAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 288x216 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VPR2paIbhM22"
      },
      "source": [
        "\n",
        "By looking at the classification matrix, we see that model has a good precision which is good in a way as it is labelling the very minimal HAM messages as SPAM. However, the recall is a bit low which suggests that there is scope for improvement in model performance.\n",
        "\n",
        "- The first approach could be hypertuning the model parameters. Selecting only those words/tokens the occur most frequently while tokenizing the data. Changing the default regularization and norm pararmeters in the logistic regression model.\n",
        "\n",
        "- The second approach would be to use more complex deep learning models but that might require more data points as these models are data hungry."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGcGwowR6nE0"
      },
      "source": [
        "## References:\n",
        "\n",
        "1. https://github.com/snorkel-team/snorkel-tutorials/blob/master/spam/01_spam_tutorial.ipynb\n",
        "\n",
        "2. https://www.snorkel.org/blog/weak-supervision\n",
        "\n",
        "3. https://medium.com/@annalara/understanding-snorkel-5e33351aa43b"
      ]
    }
  ]
}