BasicNeuralNet.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.compat.v1 as tfc # needed if using Tensorflow v2, then tfc is v1\n",
    "# https://stackoverflow.com/questions/53429896/how-do-i-disable-tensorflows-eager-execution\n",
    "tfc.disable_eager_execution() # eager execution enabled by default in Tensorflow v2\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn\n",
    "from scipy.optimize import minimize\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We consider the data of people applying for credit cards, and a banker has placed them in one of the following categories:\n",
    "\n",
    "    Approved (1)\n",
    "    \n",
    "    Denied (-1)\n",
    "    \n",
    "Each data point has 16 associated features (found in credit_data.csv) and a corresponding label (found in credit_label.csv).\n",
    "(# data = m = 653, # features = n = 16)\n",
    "\n",
    "We consider the data for two cases: data is unchanged, and data is normalized by the maximum value of each feature. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import data\n",
    "# https://realpython.com/python-csv/\n",
    "credit_data = np.loadtxt(open(\"credit_data.csv\", \"rb\"), delimiter=\",\", skiprows=1)\n",
    "credit_label = np.loadtxt(open(\"credit_label.csv\", \"rb\"), delimiter=\",\", skiprows=1)\n",
    "\n",
    "#designate training data\n",
    "training_data = credit_data[0:500, :]\n",
    "training_label = np.zeros([500,2])\n",
    "#make one-hot labels (i.e. approved ~ [1,0], denied ~ [0,1])\n",
    "for i in range(500):\n",
    "    if credit_label[i]==1:\n",
    "        training_label[i] = np.array([1,0])\n",
    "    else:\n",
    "        training_label[i] = np.array([0,1])\n",
    "\n",
    "#designate testing data\n",
    "test_data = credit_data[500:653, :]\n",
    "test_label = np.zeros([153,2])\n",
    "#make one-hot labels (i.e. approved ~ [1,0], denied ~ [0,1])\n",
    "for i in range(153):\n",
    "    if credit_label[i]==1:\n",
    "        test_label[i] = np.array([1,0])\n",
    "    else:\n",
    "        test_label[i] = np.array([0,1])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Functions we'll need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define important functions\n",
    "# Initializing the weights and biases to be randomly selected from a normal distribution with mean zero and standard deviation 1\n",
    "# These have slightly positive initial bias to break symmetry\n",
    "def weight_variable(shape):\n",
    "    initial = tfc.random_normal(shape=shape, mean=0.0, stddev=1.0)\n",
    "    return tfc.Variable(initial)\n",
    "\n",
    "def bias_variable(neurons):\n",
    "    initial = tfc.random_normal(shape = neurons, mean = 0.0, stddev = 1.0)\n",
    "    return tfc.Variable(initial)\n",
    "\n",
    "# https://stackoverflow.com/questions/40994583/how-to-implement-tensorflows-next-batch-for-own-data\n",
    "# Return a total of `num` random samples and labels. \n",
    "# makes sure Net isn't dependent on ordering of points\n",
    "def next_batch(num, data, labels):\n",
    "    idx = np.arange(0 , np.shape(data)[0])\n",
    "    np.random.shuffle(idx)\n",
    "    idx = idx[:num]\n",
    "    data_shuffle = [data[i,:] for i in idx]\n",
    "    labels_shuffle = [labels[i,:] for i in idx]\n",
    "    \n",
    "    return np.asarray(data_shuffle), np.asarray(labels_shuffle)\n",
    "\n",
    "\n",
    "# # Choose 5 random samples\n",
    "# Xtr, Ytr = next_batch(5, training_data, training_label)\n",
    "# print(Xtr)\n",
    "# print(Ytr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DATA UNCHANGED\n",
    "\n",
    "We create an artificial neural network (ANN) model with 3 layers. \n",
    "\n",
    " Layer 1: 30 neurons and linear transfer function for the activation function.\n",
    "    \n",
    " Layer 2: 30 neurons and tanh function for the activation function. \n",
    " \n",
    " Layer 3: (output layer) 2 neurons and softmax for the activation function.\n",
    " \n",
    "\n",
    "For more on activation functions: https://www.tensorflow.org/api_docs/python/tf/nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train the 3 layer Neural Network by: \n",
    "1. Initializing the weights and biases to be randomly selected from a normal distribution with mean zero and standard deviation 1\n",
    "2. Reshuffle the training data and pass through the 3 layers, then predict classes for each data point\n",
    "3. Backprop. using the Gradient Descent Optimizer, which heads in the direction of largest gradient to minimize cross-entropy loss\n",
    "4. Update the weights and biases\n",
    "5. Pass the testing data through, and make predictions, then compute the accuracy\n",
    "5. Repeat steps 2-5 for 1000 iterations.\n",
    "\n",
    "We test the data every 100 iterations using the testing data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.53\n",
      "test accuracy 0.366013\n",
      "step 50, training accuracy 0.672\n",
      "step 100, training accuracy 0.68\n",
      "test accuracy 0.503268\n",
      "step 150, training accuracy 0.68\n",
      "step 200, training accuracy 0.68\n",
      "test accuracy 0.503268\n",
      "step 250, training accuracy 0.682\n",
      "step 300, training accuracy 0.682\n",
      "test accuracy 0.503268\n",
      "step 350, training accuracy 0.676\n",
      "step 400, training accuracy 0.676\n",
      "test accuracy 0.503268\n",
      "step 450, training accuracy 0.682\n",
      "step 500, training accuracy 0.682\n",
      "test accuracy 0.503268\n",
      "step 550, training accuracy 0.682\n",
      "step 600, training accuracy 0.68\n",
      "test accuracy 0.542484\n",
      "step 650, training accuracy 0.68\n",
      "step 700, training accuracy 0.682\n",
      "test accuracy 0.54902\n",
      "step 750, training accuracy 0.684\n",
      "step 800, training accuracy 0.684\n",
      "test accuracy 0.54902\n",
      "step 850, training accuracy 0.688\n",
      "step 900, training accuracy 0.69\n",
      "test accuracy 0.54902\n",
      "step 950, training accuracy 0.69\n",
      "step 1000, training accuracy 0.692\n",
      "test accuracy 0.542484\n"
     ]
    }
   ],
   "source": [
    "#start an interactive session\n",
    "sess = tfc.InteractiveSession()\n",
    "\n",
    "#tf.enable_eager_execution()\n",
    "\n",
    "\n",
    "#placeholders for input images and 1 hot class vector\n",
    "#inputs are 1x15 feature vector\n",
    "#note that the shape is optional, but will allow us to catch bugs later\n",
    "x = tfc.placeholder(tfc.float32, shape=[None, 15])\n",
    "y_ = tfc.placeholder(tfc.float32, shape=[None, 2])\n",
    "\n",
    "\n",
    "#define weights and biases, initialized as zeros for first hidden layer with 30 nodes\n",
    "#note that the default is to save a variable as tf.float32\n",
    "W_layer1 = weight_variable([15,30])\n",
    "b_layer1 = bias_variable([30])\n",
    "\n",
    "\n",
    "#define the class predictions\n",
    "######## Activation relu \n",
    "h_layer1 = tfc.matmul(x,W_layer1) + b_layer1\n",
    "\n",
    "#define weights and biases, initialized as zeros for second layer \n",
    "#note that the default is to save a variable as tf.float32\n",
    "W_layer2 = weight_variable([30,30])\n",
    "b_layer2 = bias_variable([30])\n",
    "\n",
    "#define the class predictions\n",
    "######## Activation tanh\n",
    "h_layer2 = tfc.nn.tanh(tf.matmul(h_layer1,W_layer2) + b_layer2)\n",
    "\n",
    "#define weights and biases, initialized as zeros for second layer \n",
    "#note that the default is to save a variable as tf.float32\n",
    "W_layer3 = weight_variable([30,2])\n",
    "b_layer3 = bias_variable([2])\n",
    "\n",
    "#define the class predictions\n",
    "######## Activation softmax\n",
    "h_layer3 = tfc.nn.softmax(tfc.matmul(h_layer2,W_layer3) + b_layer3)\n",
    "\n",
    "#define the cross entropy loss\n",
    "#the cross entropy logits function applies a softmax on the model's unnormalized prediction and\n",
    "#sums across all classes\n",
    "#the reduce_mean function takes the average over the sums\n",
    "cross_entropy = tfc.reduce_mean(tfc.nn.softmax_cross_entropy_with_logits_v2(labels=y_, logits=h_layer3))\n",
    "\n",
    "#define a training step, which you will run later with the train_step.run command\n",
    "#train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)\n",
    "train_step = tfc.train.GradientDescentOptimizer(learning_rate = 0.1).minimize(cross_entropy)\n",
    "\n",
    "# #define a true/false vector to evaluate predictions\n",
    "correct_prediction = tfc.equal(tfc.argmax(h_layer3,1), tfc.argmax(y_,1))\n",
    "\n",
    "# #recast the true/false vector as a binary vector and take the average\n",
    "accuracy = tfc.reduce_mean(tfc.cast(correct_prediction, tfc.float32))\n",
    "\n",
    "\n",
    "#add a tensorboard graph\n",
    "writer = tfc.summary.FileWriter('./nn_graph', sess.graph)\n",
    "# add a summary to store the accuracies\n",
    "training_summary=tfc.summary.scalar('train_acc', accuracy)\n",
    "testing_summary=tfc.summary.scalar('test_acc', accuracy)\n",
    "\n",
    "#initialize all variables\n",
    "sess.run(tfc.global_variables_initializer())\n",
    "#np.random.seed(1)\n",
    "\n",
    "train_accuracy = np.zeros(1001)\n",
    "test_accuracy = np.zeros(1001)\n",
    "\n",
    "#run 1000 iterations of SGD with batch size equal to 500\n",
    "for i in range(1001):\n",
    "    batch = next_batch(500, training_data , training_label)\n",
    "    \n",
    "    train_accuracy[i],summary = sess.run([accuracy,training_summary],feed_dict={x:batch[0], y_: batch[1]})\n",
    "    writer.add_summary(summary, i)\n",
    "    \n",
    "    test_accuracy[i],summary = sess.run([accuracy,testing_summary],feed_dict={x: test_data , y_: test_label})\n",
    "    writer.add_summary(summary, i)\n",
    "    \n",
    "    if i%50 == 0: #evaluate and print the training accuracy every 10 iterations\n",
    "        print(\"step %d, training accuracy %g\"%(i, train_accuracy[i]))\n",
    "    if i%100 == 0: #evaluate and print the testing accuracy every 100 iterations\n",
    "        print(\"test accuracy %g\"%accuracy.eval(feed_dict={x: test_data, y_: test_label}))\n",
    "\n",
    "    train_step.run(feed_dict={x: batch[0], y_: batch[1]})\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Un-normalized data accuracy')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeUAAAEYCAYAAABiPO6lAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwYElEQVR4nO3de5xVdb3/8dd7ZrgIImCAGIqgYop1vI140BRKLC1LBetHHk920dHM0n6dvNTpaJ0uaBftd7JkjqVplqcTaB4rVCjQOliOCioXFQ0VAQFvXAJhmM/vj7W2bDd7hj2XPXvNzPv5eOzH3mut73et73fPnv3Z38taSxGBmZmZVV5VpQtgZmZmCQdlMzOzjHBQNjMzywgHZTMzs4xwUDYzM8sIB2UzM7OMcFDOIEk3S1rRzLaJkkLSpM4uV1chabmkm/OWP5G+Z6MqVYZW5r1KUqvPVZQ0Ks27f1uOa2aV56BsPcFvgfHAqkoXpMxGAVcCDspmXVRNpQtg3Z8kAb0iYmsljh8Ra4G1lTi2dR5JfSLijUqXw6w93FLuBtKu0p9LmippiaRNkhokvbuj80s6W9JCSVskrZN0q6S9m9nfpyQtBbYCH8zrRj5W0q8kbZD0kqQr0nwnS3o0Pf5Dko4q2O/7JP1O0ipJf5f0hKQvSqreRf3e0n2dDg9EM4+JefkOk3SXpFclbZb0Z0nHF9n/xWmdt6Tv205pWijbEZIeSPO+KOmrgIqku0jSfEmvSHpN0oOSPpi3fSLwx3TxvsL6pH/bP0haK2lj+j6fU2IZS8orqUbSZZIWp/VZK2mWpIPz0gyV9CNJL0h6I32+VVKfdPvNkpYX2fdcSXPz65vWb7Kk/5S0Fngp3XZgus+/pX+3ZyX9WNLgIvudIOk+Sa+nn7uFkj6dbrtb0iNF8oyW1CTp/FLeP7PWcEu5+zgeeAfwVWAL8O/A3ZJGRcRrHZFfUh0wHfgv4Arg7cC3gGMkHRkRG/P29x7gcOBrwBpgeXoMgJ8BtwD1wEeAb0kaBHwA+CawEbgGuFPSAXkt7P2BOcB/pGWsBa4ChgKXl1DHnH8HbihY913gCOD5tK5HAg8AjwLnAX8HLgBmSzo2Ih5O030auA64OX1fDgR+CQzYVSEkDQH+AKwGzgHeAL4EjCySfBRwI8n7WAN8iOTv84GI+D3wCPBZ4Hrg88BDab7F6fP+wK+BaUATcAJwo6TdIqLwvShUat7bgdNJ3o/ZQN807d7A0jQo/i+wJ/AN4DFgGHAa0Dutf2v9B/B74J/T40HyuVwBXAK8mpb/y8DvSIYxAJB0GjAD+DNwPrAOOBTYL03yI+C3ksZFxF/zjlkHbAJ+0YbymrUsIvzI2IPkC35FM9smAgFMylu3nOTLZ3Deuto03VklHG+X+YFqkpbIHwvyvjtN9/mC/f0dGF6Q9hNp2n/LW1dDErS3AaPz1n84TTuhmTIrzfuVtOxVBce/uchxRzWzr38BtgOn562bAywBeuetq07X3ZkuVwEvALMK9vd/0uPdXOx4eem+SdKLMDJvXX+S4BAt5KtK634v8JuWPhu7yP+fwMJWfjaL5gXeW/g5KJL36+n7fMQuPvvLi6yfC8wtUtc7SihzTd7n9Ii8z89yoCH/s1Okrs8AP8lb14vkR9QNrXnf/PCj1Ie7r7uP+RHxat7y4+nzSEjGddPuxdyjsMu3xfwkrehhwG35mSLiT8BzwISC/T0YEaubKevv8/I3AsuApyLib3lplqbP++ZWSNpb0nRJz5EEs20kLa5BadlaTdKHgKuByyLiznTdbml9/htoyr1nJF/ks0lafwD7pI9fFex2BtBYwuHHk7xPz+dWRMQm4H+KlPOotDv1pXTf24CTSP4updRzjKRfSnoxzbsNOLeU/CXmfR9J0PvPFnb1PuChiHi0lDKX6I4i5e0t6cuSlkranJb3gXTzO/Ke9wNujIimYjtO108HpkoamK4+HdgrXW/W4RyUs6mRpFVWTHVemnyv5C/EjgkvuS69c9jxhbqNpAXQmvx7ps/FZjCvzttOC+lyXi1Y3trMujePL6kKuAs4lSQQvxc4mqS1mV/Okkk6jKQL8icR8d28TXuSvM9f5a3v2TbgImBwWp7cWPpL+ftNf2i8XEIR9i7MW2x/kvYlabnvCXwOOJak7rMood6SdgfuAw4j6eY/Ps3/U6BPB+V9G/BKRGxuYXdvI+lW7kjFPmffJhnW+DnwQWAcMDndlnu/3pY+76o8PyH5nvzndPkC4K8d/MPC7E0eU86mNcAQSb1j5xnLb0+fi32Zt+R/SL5Mc1o7fpcL2sOLbBtO0g2Yr6PvCXoASZf6P0fEz3Mr05Zuq0naiyTIPwhcWLD5NZKx0+tJxr53EhFNknIBYa+Cfdew40u/JasK8xbbH3AyMBD4aES8GUQk9SvhGJC0yPcDjk97NvLL2VF51wF7puPMzQXmdcCIXRxvC8n4cqG3UfyHTrHP2VTgloj4Rl55dy9SFnZVnoh4WdJ/A+dLuodkrsS5LeUxaw+3lLPpjyQ/mD5cZNsUki/zJ1uzw4h4OSIa8h6P7zrXWzxJ8kNgav5KSceSfGnPa+X+WisXgLblHbsX8E+t3ZGkvsBvSCbrfCRt2b4p7UJ+gKR1+EjB+9YQEbkfICtIxpQ/WnCIKZT2g3c+8I9pSzhXtv4kk7jyFav7QcBxBelyP7R2KyH/YJIJVrtSat57Sbr3WwpY9wLj0h6K5jwH7JVOgssd7wBK7KbPK/O2gnWfLFh+imRM+VxJO812L/Aj4J0kE+3Wk0xoMysLt5SzaTZJl+HN6ekkfyGZzTuV5Mvwk82Ng5VLRGyX9G/AdEk/J+kaHEHSffw0cFOZi7CE5Av7m5K2k3zpfqGN+7oOOJJkAtjBBd/JiyNiPfB/gfuBeyT9hOSH0JA0X3VEXJ62lr9GMhP5JpIv6wNJZqavL6Ec15K00u+VdBU7Zl8XtjRnkwxX3CLpeyTd3l8jmSme/8P6qTTdpyS9ku7vSZIZz+uB6yVdSTKZ7F9JWosDaVlJeSPij5JmAN9Pf2T8gWRS1AnAbyNiblrfs0hmsH+DZN7CEJLP9AURsYFkHP/fgdskfT/dfgU7WralmAWcI+lxkvkKk0m6/N8UESHpEmAm8AdJN5Ccy34IMCwirsxL+2B6atQJwH9ExN9bURaz1qn0TDM/ij9Ixr6+QfJF+wawgaT1dlqRtMuBnxdZH8BVJRyr5PzA2cDCtEwvA7cCe5e4v0+k+zywYP1c4E8F60alac/NW3c48CeSmd0rSGbznkvBzGp2Mfs6PV4085iYl+8QkkC7Jq3vCpIu7w8UlPVikh8MW0i68d9dWIYW3vvcqVdbgBdJxrG/RsHsa5LW+NI03SKSH2g3UzBTmeTUnmdJgvOb9SEZg3+UJOA/Q3La1FWFx2mmjCXlZcds+KdI5gSsJTkN6R15aYaRnAq3Kk3zAskpcn3y0pwOPJEebyHJBLG5FJ99vdNMc5JAfjvJPIVXSSYnHp2m/0SRuv2R5DS8jenxPllkn1ek+Q+t9HeDH937oYiOHvozM+teJP0ZaIqIki8MY9YW7r42MytCyVXGjgQmkXR/lzIGb9YuDspmZsXtTTKm/hrwrYi4q7LFsZ7A3ddmZmYZ4VOizMzMMqJbdV8PGTIkRo0aVelimJl1GQ8//PC6iBha6XJYolsF5VGjRtHQUHhhKTMza056LXnLiLJ2Xyu5P+6TkpZJ2unWepK+JGlB+nhC0nZJe5aS18zMrLspW1BO70J0PXAKMBb4mKSx+Wki4jsRcXhEHE5ycv68iHillLxmZmbdTTlbyuOAZRHxbCQ3Vbidls/z+xjJzeHbktfMzKzLK+eY8giSS+jlrACOKZYwvdvNySS3xWtVXjMzy4aHH354WE1NzY0kN/Dw2T3FNQFPNDY2nnvUUUetKdxYzqBc7M4rzZ0U/SHgzxGRuz1gyXkl1QF1ACNHjmxtGc3MrIPU1NTcOHz48EOGDh36alVVlS+CUURTU5PWrl07dvXq1TdS5E6A5fwlswLYN295H2BlM2mnsqPrulV5I6I+ImojonboUM/qNzOroHcOHTp0vQNy86qqqmLo0KGvk/Qm7Ly9jMd+CBgjabSk3iSBd6fL1EkaCEwgub9tq/KamXUn8+fDGWfA3ntD375QUwO9eiWP3Os+fXZeHjwYLrus0qUHoMoBedfS96ho/C1b93VENEq6CLgHqAZ+GhGLJF2Qbr8hTXoGcG8kN5ZvMW+5ympm1pL6erjySliXd1fnCJCgqgqamnYst3VbBGzf3rbybd0K11yTvL766rbX0yqvrAPxEfG7iDgoIg6IiG+m627IC8hExM0RMbWUvGZmna2+Hs4/H1avhsbGHY/t25PnrVvfutzWbW0NyPlmzmz/PrqydevWVU+bNq3V45gTJkw4cN26ddUtpbnkkkvefueddw5oe+lK49lxZtZjzZ8PEybAgAE7dwvnXl9wQaVLWbrJkytdgsp6+eWXq3/yk58MK1zf2NjYYr558+YtGzJkSIs/i6677rqVp59++oZ2FnGXHJTNrFmXXQZjxmRmvLJDzZ8Pxx8P998PGzc236qt5I30qqqSHwY1NVBdnTz37r3z8qBBcOmlXbPrevZs+l9xBcNnz6Z/e/f1xS9+cZ8XXnihz8EHHzz2ne985yHHHHPMQR/60IdGv+Md7zgUYNKkSQcceuihhxx44IGHfve73x2SyzdixIh3rVq1qubJJ5/svf/++x86derU/Q488MBDjzvuuDEbN24UwJQpU0bddNNNg3Ppv/CFL7x97Nixhxx00EFjH3300b4AK1eurDn22GPHjB079pCzzjprv7e//e3vWrVqVauGiR2UzbqY+fPh299OulVzrbxyTPS57LJknHLZsuT57LM7Zr+ltE6LTWjq6G3vfnfbuoxzgbK5INmebTU1sOeecPrp8L//m5Rv27bk0diYPL/xxs7Lr77adQPyqady0DXXMOLUUzmovYH5e9/73op99933jaVLly6eNm3aiscee6z/d77znRefeeaZRQC33Xbb8kWLFi1ZsGDB4unTp++1evXqnbqsn3/++b6f//zn1yxbtmzRwIEDt99yyy2Dix1ryJAhjYsXL17yqU99au20adP2Arj88svfPmHChA2LFy9eMnny5FdXrVrVu7V16FY3pLCOVV8P3/oWrF2btCLaO5GlNdt694Yjj4Rp02D8+M6ve7758+Hyy+GRR2DLlvLXfVfvy+bNxYPJNdfA974HU6fCz3+elHvuXJg4sW3vYeH45G23wS9/mZSjNXXIX4aOGTutlJqapGVd6c9kdzFnDgO2baOqqQkaG6maM4cBkyaxadc5S/MP//APmw4++OCtueWrr756r9/+9reDAFavXt1r0aJFfYcPH/6W440YMeKNY489djPAEUcc8ffly5f3Kbbvs84661WAcePG/f2uu+4aDPDXv/519zvvvHMZwJlnnrl+jz32aPWn3UG5Qi67DKZPh02bOvdLvdRtTU3Jo1K2bk2+/E44obJfgrkuzqwEkq1bW96+fXsSPG+77a3rq9P2QGs+E8X+/pX+XJRbTU3x96Jv3+z8SOxOTjyRDddeS1NjI1U1NTSdeCIdOmbbr1+/Nz+td99994B58+YNaGhoWDpgwICmcePGvWPz5s079Rb37t37zQGL6urqKJYGoG/fvgFQU1MTjY2NAogOGOtwUK6AXLeg7VpjY9Laq9QX4dy52QnI7dEd6lBuXXVMtiubNIlNd9/NU3PmMODEE9nQ3lbywIEDt2/atKloEH3ttdeqBw4cuH3AgAFNjz76aN+FCxe2ewy70Lhx4zbeeuute37zm99cPXPmzD3Wr1/f4ozuYhyUK6Cnn7bQGjU1SfdrpUycmLQyHdTKI9cVXqkeoaoq6NcP6uockCtl0iQ2dVSX9fDhw7cfddRRG8eMGXNonz59moYOHbott23KlCmv19fXDz3ooIPGHnDAAVsOO+ywDusmz5k2bdrKM888c/+xY8cOHj9+/MahQ4duGzRoUKu+PdQRze2sqK2tjYaGhkoXYye5rurNm5MvhF3Mzs+ktnR/elvHbmtpJvC4ccmYd7k+W717J2VobGxb4HMgzC5JD0dEbUfsa+HChcsPO+ywdbtO2T1t3rxZNTU10atXL2bPnt3/oosu2m/p0qWLi6VduHDhkMMOO2xU4Xq3lAvU18OMGTBlSvKlUarmJgPtagwu9yVV6S/8YtuqquCQQ+DHP+7c7uMsd+9Xuovz7LPhrruSWcRbtiQ9CfnBLX+uAiTBFFo/UQ929A706lXZIQSzrmLZsmW9P/rRjx7Q1NREr169Yvr06ctbuw+3lPPkrtyTkwuWuwpo0PYWyoEHwtNPt7nI3dKYMclpOFnUk/5e7Z29bV2DW8qV4ZZyCa677q3LnTHLtKdfgaeYyZOz21LuSX+v8eMdjM06m4Nynlyrt5xyY7O77QYXXugxtWJy70n+OHwlu/XBY6Bm1jkclPNcfPFbu6/bKn+cOGsXwugqrr7aAdDMeh4H5Ty5iV3FrmLl1pSZmZWbr31doK4Oli9PZq+2dL3Zwm1d+fqzZmbdQVtv3Qjw9a9/fdiGDRvejIml3M6xHByUzcysW2ju1o2lmD59+l4bN258MyaWcjvHcnD3tZmZVc7s2f2ZM2cAJ564gUmT2nWVrfxbN06YMGH9sGHDtt1xxx17bt26VR/84Adfu/baa1euX7++6sMf/vD+q1at6t3U1KRLL7105UsvvdRrzZo1vSZMmHDQ4MGDG//yl788NWLEiHc1NDQsWb9+fdUpp5wyZty4cRsbGhp232uvvbbec889y3bfffeYN29ev/POO29Uv379mo455piNf/jDHwY+/fTTi9pTB7eUzcysMmbP7s+ppx7ENdeM4NRTD2L27A67deP73//+9cuWLev72GOPLVmyZMniBQsW9Pv973+/+8yZM/cYPnz4tieffHLx008/vWjy5Mnr//Vf/3XNsGHDts2bN++pv/zlL08V7re52zmee+65o6+//vrnFixYsLS6urpDLvrhoGxmZpUxZ84Atm2rSq8/XMWcOQM6atezZs3a4/77799j7NixYw899NCxzzzzTN+lS5f2PfLIIzc/8MADe3zmM58ZMWvWrN3f9ra37bKLutjtHNetW1e9adOmqpNOOmkTwDnnnPNKR5Tb3ddmZlYZJ564gWuvbaKxsYqamiZOPLHDbt0YEVxyySWrvvSlL+10hbFHHnlk8YwZMwZ+5StfGTF79uz13/3ud1e1tK9it3Ms19Uw3VI2M7PKmDRpE3ff/RRf+tKL3H33U+0dU86/deMpp5yy/tZbbx3y+uuvVwH87W9/6/Xiiy/WLF++vNeAAQOaLrzwwlcuueSSlxYsWNAPoH///ttzaUsxdOjQ7f3792+aM2dOf4Bbb711z/aUPaesLWVJJwM/AKqBGyNiWpE0E4HrgF7AuoiYkK5fDmwAtgONHXVtVjMzy5BJkza1Nxjn5N+68b3vfe/rH/nIR145+uijDwbo169f02233fa3pUuX9rniiiv2qaqqoqamJn70ox89B3DOOeesO+WUU8YMGzZsW7Fx5WKmT5++/IILLtivX79+Tccdd9yGAQMGtHu2dtluSCGpGngKOAlYATwEfCwiFuelGQT8L3ByRDwvaVhErEm3LQdqI6Lki5tn9daNZmZZ5RtStN3rr79eNXDgwCaAL3/5y8NXrVrV66abbnqhlLyVuCHFOGBZRDwLIOl24DQg/96SZwEzI+J5gFxANjMzy7pf/epXA7/3ve/tvX37do0YMeKNX/ziF8vbu89yBuURQP4vhhXAMQVpDgJ6SZoLDAB+EBG3pNsCuFdSANMjor7YQSTVAXUAI0eO7LjSm5mZteC888579bzzznu1I/dZzqBc7J5LhX3lNcBRwInAbsB8SQ9GxFPAcRGxUtIw4D5JSyPi/p12mATreki6rzu0BmZm1hpNTU1Nqqqq8ndxC5qamgQUvTlwOWdfrwD2zVveB1hZJM2siNiUjh3fDxwGEBEr0+c1wB0k3eFmZpZdT6xdu3ZgGnSsiKamJq1du3Yg8ESx7eVsKT8EjJE0GngRmEoyhpzvN8APJdUAvUm6t6+V1B+oiogN6ev3AV8vY1nNzKydGhsbz129evWNq1evfic+5bY5TcATjY2N5xbbWLagHBGNki4C7iE5JeqnEbFI0gXp9hsiYomkWcBjaUFvjIgnJO0P3KHkHok1wC8iYla5ympmZu131FFHrQE+XOlydGVlOyWqEnxKlJlZ63TkKVHWfu5eMDMzywgHZTMzs4xwUDYzM8sIB2UzM7OMcFA2MzPLCAdlMzOzjHBQNjMzywgHZTMzs4xwUDYzM8sIB2UzM7OMcFA2MzPLCAdlMzOzjHBQNjMzywgHZTMzs4xwUM4zfz58+9vJs5mZWWerqXQBsmL+fJgwAbZtg169YN48GD++0qUyM7OexC3l1DXXJAEZkudrrqlseczMrOdxUE6tXNnyspmZWbk5KKc+/emWl83MzMrNY8qpurrkecYMmDJlx7KZmVlnKWtLWdLJkp6UtEzS5c2kmShpgaRFkua1Jm9Hq6uDe+5xQDYzs8ooW0tZUjVwPXASsAJ4SNJdEbE4L80g4EfAyRHxvKRhpeY1MzPrbsrZUh4HLIuIZyNiK3A7cFpBmrOAmRHxPEBErGlFXjMzs26lnEF5BPBC3vKKdF2+g4DBkuZKeljSx1uRFwBJdZIaJDWsXbu2g4puZmbW+co50UtF1kWR4x8FnAjsBsyX9GCJeZOVEfVAPUBtbW3RNGZmZl1BOYPyCmDfvOV9gMKzf1cA6yJiE7BJ0v3AYSXmNTMz61bK2X39EDBG0mhJvYGpwF0FaX4DHC+pRlI/4BhgSYl5zczMupWytZQjolHSRcA9QDXw04hYJOmCdPsNEbFE0izgMaAJuDEingAolrdcZTUzM8sCRXSfYdja2tpoaGiodDHMzLoMSQ9HRG2ly2EJX2bTzMwsIxyUzczMMsJB2czMLCMclM3MzDLCQdnMzCwjHJTNzMwywkHZzMwsIxyUzczMMsJB2czMLCMclM3MzDLCQdnMzCwjHJTNzMwywkHZzMwsIxyUzczMMsJB2czMLCMclM3MzDLCQdnMzCwjHJTNzMwywkHZzKyS6uth1Cjo2xf69IHDD4f58ytdKquQkoKypBmSPijJQdzMrKPU18P558Nzz8Ebb8DWrbBwIRx/vANzD1VqkP0xcBbwtKRpkg4uJZOkkyU9KWmZpMuLbJ8o6XVJC9LHv+VtWy7p8XR9Q4nlNDPrOmbMKL5++3aYO7dTi2LZUFNKooiYDcyWNBD4GHCfpBeA/wR+HhHbCvNIqgauB04CVgAPSborIhYXJH0gIk5t5tDviYh1JdbFzKxrmTIF7r135/XV1TBxYqcXxyqv5O5oSW8DPgGcCzwK/AA4ErivmSzjgGUR8WxEbAVuB05rV2nNrPIuuwwGDYJevaCmJnnOf92nT8vLzW0bMCDZd3vkj8+WUrZKb/vsZ5MAXFOTPHKvd9sNLr/cXdg9UEktZUkzgYOBW4EPRcSqdNN/tdC1PAJ4IW95BXBMkXTjJS0EVgL/EhGL0vUB3CspgOkRUd9M2eqAOoCRI0eWUh0za6vLLoNrrinPvjdu3LHvq69uff7c+GxXJUFE8nrjRrj/fjjhhOR5/PjKls06Takt5R9GxNiI+HZeQAYgImqbyaMi66Jg+RFgv4g4DPgP4M68bcdFxJHAKcBnJZ1Q7CARUR8RtRFRO3To0FLqYmZtNXNmdo/R3PhsVxGFX49AY6PHlnuYUoPyIZIG5RYkDZZ04S7yrAD2zVveh6Q1/KaIWB8RG9PXvwN6SRqSLq9Mn9cAd5B0h5tZfT28//1Jq3XChKTbt08f6NcP9tgD+vd/azdpv34wenSSb/58OOII6N279K7X/NN0Jk8uf/2WLWtbt/B9zY2kdREq1o4BvvrVXQ8PDB7c/q5/ywRFsV9nhYmkBRFxeMG6RyPiiBby1ABPAScCLwIPAWfldU8jaTjwUkSEpHHAr4H9gH5AVURskNSfZNz66xExq6Vy1tbWRkODJ2pbN1bJLtrqanjgAbjzTpg+HTZtSlp3uWCSe11VBU1NzS8XS9vY2PHlrapKHi2VLQvbqqrgkEPgxz+Gxx+HK66AV15pW50vvbTVXf+SHm6hx9M6WUljykCVJEUawdOZ1b1byhARjZIuAu4BqoGfRsQiSRek228AzgQ+I6kR2AxMTQP0XsAdSj7ENcAvdhWQzXqESnbR5k7Tufrqto35tmTMmKSF3JG+8Y0kwHUl48cnf+NiM7JLMXNmx/9trFOVGpTvAX4l6QaSceELgF0GybRL+ncF627Ie/1D4IdF8j0LHFZi2cx6juZOoekM5TxNZ/Lkjp1AVlPTdU8pas/fuDOGF6ysSg3KlwHnA58hmcB1L3BjuQplZs2oq0ueZ8xIxnkffBAeeSS5ElRndL0ef3xyus6FF3Zsiyy3r+uvhy1b2l7u3r3h6KNh2rSuO2M59ze+8kpYl3eZhpaGB/r1S/K5ldzllTSm3FW0dUy5vh6uuy75fF988Y7/CbMuo3Csefr05IM8fz68+93Jl3ihXJrW7DdfG8YvLXs8ppwtpV77eoykX0taLOnZ3KPchesMue+cJUtg8eLkdX3RM6LNMqxwrDm3PHdu8YBcLE9r03TG6VFmPUypp0TdRHL960bgPcAtJBcS6fKKfed09dMdrQeaMqX48sSJO2Yg7ypPa9N4/NKsw5UalHeLiDkk3d3PRcRVwHvLV6zOU+w75/DDO70YZu1TV5d0R7/vfW/tlh4/Hv70p+RD3atXMuY6alRpXdf5+91vv+R82Opq2H13d12blUmpE722pLdtfDo9zelFYFj5itV53vWuHfMmIBlXHjSookUya5u6uuKBdvx4ePTRjt+vmXW4UlvKl5Bc0OPzwFHA2cA5ZSpTpyq8gl1XPpPCzMy6tl22lNMLhXw0Ir4EbAQ+WfZSdaKJE5NeuTfeSFrMP/xh1z2TwszMurZdBuWI2C7pqPwrenUn48fDnDlJi3niRAdkMzOrnFLHlB8FfiPpv4FNuZUR0S3OiRg/3sHYzMwqr9SgvCfwMm+dcR1AtwjKZmZmWVBSUI6IbjWObGZmlkUlBWVJN5G0jN8iIj7V4SUyMzProUrtvr4773Vf4AxgZccXx8zMrOcqtfv6LReelPRLYHZZSmRmZtZDlXrxkEJjgJEdWRAz24X58+GII2DgQDj77GT5299Ons2sWyh1THkDbx1TXk1yj2Uz6wyFt2C87Ta4/fbkde/eycn2Pq/PrMsrtft6QLkLYmYtKHYLxu3bk+etW5PtDspmXV6p91M+Q9LAvOVBkk4vW6nM7K2K3YKxujp59O7tC7abdROlzr6+MiLuyC1ExGuSrgTuLEupuqL6evjWt2Dt2qTlEpHccip3C6rcMnibt7VtW01Nsq2qCvbYA7ZsSdbV1bmVbNZNlBqUi7WoS7mZxcnAD4Bq4MaImFawfSLwG+Bv6aqZEfH1UvJmSn09nH9+pUthPcm6dTteX3MNHHCAb69o1g2UOvu6QdL3JR0gaX9J1wIPt5QhvbvU9cApwFjgY5LGFkn6QEQcnj6+3sq82TBjxq7TmJWTP4Nm3UKpQflzwFbgv4BfAZuBz+4izzhgWUQ8GxFbgduB00o8Xnvydr4pUypdAuvp/Bk06xZKnX29Cbi8lfseAbyQt7wCOKZIuvGSFpJcIexfImJRK/IiqQ6oAxg5sgNOna6vT1odU6YU7w6cPz/pLnz00WScr1cvWLUqGdvLwrijt3Xfbb16wV57wUc/Cg8+CI89BoMGwRVXuOvarJso9Tzl+4CPRMRr6fJg4PaIeH9L2YqsK7x+9iPAfhGxUdIHSCaOjSkxb7Iyoh6oB6itrW3f/Z7zx4bvvTd5zv+ymz8fTjgBGhtb3s/06f6SNDOzViu1+3pILiADRMSrwLBd5FkB7Ju3vA8F18uOiPURsTF9/Tugl6QhpeQti8JxucLluXN3HZCL5TMzMytBqUG5SdKbfcOSRtFMyzXPQ8AYSaMl9QamAnflJ5A0XEr66CSNS8vzcil5y6JwXK5weeLEpJu6tfsxMzMrQamnRH0F+JOkeenyCaTjuM2JiEZJFwH3kJzW9NOIWCTpgnT7DcCZwGckNZJMHpsaEQEUzdvKurVeXR3cfz/8+tfJ1ZI++1m48MKdx/pygTl/HLC6Ohnv8/iemZm1kZIYWEJCaRhJIF5AcvvGNRFxf/mK1nq1tbXR0NDQ9h2053xjjyObWRck6eGIqK10OSxR6mU2zwXmAF9MH7cCV5WvWBXSnrFgjyObmVk7lTqmfDFwNPBcRLwHOAJYW7ZSVUp7xoI9jmxmZu1U6pjylojYIglJfSJiqaR3lLVklZDrfm7pGtaQXHd45Eh49VXo3x8uvthd12Zm1m6lBuUVkgaRnEd8n6RX6YxTlCqhrs4B1szMKqLUK3qdkb68StIfgYHArLKVyszMrAcqtaX8poiYt+tUZmZm1lqlTvQyMzOzMnNQNjMzywgHZTMzs4xwUDYzM8sIB2UzM7OMcFA2MzPLCAdlMzOzjHBQNjMzywgHZTMzs4xwUDYzM8sIB2UzM7OMcFA2MzPLCAdlMzOzjHBQNjMzy4iyBmVJJ0t6UtIySZe3kO5oSdslnZm3brmkxyUtkNRQznKamZllQavvp1wqSdXA9cBJwArgIUl3RcTiIumuBu4pspv3RMS6cpXRzMwsS8rZUh4HLIuIZyNiK3A7cFqRdJ8DZgBrylgWMzOzzCtnUB4BvJC3vCJd9yZJI4AzgBuK5A/gXkkPS6pr7iCS6iQ1SGpYu3ZtBxTbzMysMsoZlFVkXRQsXwdcFhHbi6Q9LiKOBE4BPivphGIHiYj6iKiNiNqhQ4e2q8BmZmaVVM6gvALYN295H2BlQZpa4HZJy4EzgR9JOh0gIlamz2uAO0i6w8vrsstgzJjk2czMrJOVbaIX8BAwRtJo4EVgKnBWfoKIGJ17Lelm4O6IuFNSf6AqIjakr98HfL2MZU0C8TXXJK9zz1dfXdZDmpmZ5StbSzkiGoGLSGZVLwF+FRGLJF0g6YJdZN8L+JOkhcBfgd9GxKxylRWAmTNbXjYzMyuzcraUiYjfAb8rWFdsUhcR8Ym8188Ch5WzbDuZPHlHCzm3bGZm1onKGpS7lFxX9cyZSUB217WZmXUyRRROiO66amtro6HBF/8yMyuVpIcjorbS5bCEr31tZmaWEQ7KZmZmGeGgbGZmlhEOymZmZhnhoGxmZpYRDspmZmYZ4aBsZmaWEQ7KZmZmGeGgbGZmlhEOymZmZhnhoGxmZpYRDspmZmYZ4aBsZmaWEQ7KZmZmGeGgbGZmlhEOymZmZhnhoGxmZpYRDspmZmYZUdagLOlkSU9KWibp8hbSHS1pu6QzW5vXzMysuyhbUJZUDVwPnAKMBT4maWwz6a4G7mltXjMzs+6knC3lccCyiHg2IrYCtwOnFUn3OWAGsKYNec3MzLqNcgblEcALecsr0nVvkjQCOAO4obV58/ZRJ6lBUsPatWvbXWgzM7NKKWdQVpF1UbB8HXBZRGxvQ95kZUR9RNRGRO3QoUNbX0ozM7OMqCnjvlcA++Yt7wOsLEhTC9wuCWAI8AFJjSXmNTMz61bKGZQfAsZIGg28CEwFzspPEBGjc68l3QzcHRF3SqrZVV4zM7PupmxBOSIaJV1EMqu6GvhpRCySdEG6vXAceZd5y1VWMzOzLFBE0aHaLqm2tjYaGhoqXQwzsy5D0sMRUVvpcljCV/QyMzPLCAdlMzOzjHBQNjMzywgHZTMzs4xwUDYzM8sIB2UzM7OMcFA2MzPLCAdlMzOzjHBQNjMzywgHZTMzs4xwUDYzM8sIB2UzM7OMcFAGqK+H978/eTYzM6uQct5PuWuor4fzz09e33tv8lxXV7nymJlZj+WW8owZLS+bmZl1EgflKVNaXjYzM+sk7r7OdVVfdx1IFS2KmZn1bA7KOUuWJM+58WWPK5uZWSdz9zV4XNnMzDLBQRk8rmxmZplQ1qAs6WRJT0paJunyIttPk/SYpAWSGiS9O2/bckmP57aVs5y8611wwgmwzz5w6aXuujYzs4oo25iypGrgeuAkYAXwkKS7ImJxXrI5wF0REZL+AfgVcHDe9vdExLpylRGA+fNh4kTYujVZ/sEP4PTTYfz4sh7WzMysUDlbyuOAZRHxbERsBW4HTstPEBEbIyLSxf5A0NnmzoVt23Ysb92arDMzM+tk5QzKI4AX8pZXpOveQtIZkpYCvwU+lbcpgHslPSypfP3JEydCr147lnv3TtaZmZl1snKeElXspN+dWsIRcQdwh6QTgH8HJqWbjouIlZKGAfdJWhoR9+90kCRg1wGMHDmy9aUcPz5pGd9yS7L88Y+769rMzCqinEF5BbBv3vI+wMrmEkfE/ZIOkDQkItZFxMp0/RpJd5B0h+8UlCOiHqgHqK2tbVv39/jxDsRmZlZx5ey+fggYI2m0pN7AVOCu/ASSDpSSy2hJOhLoDbwsqb+kAen6/sD7gCfKWFYzM7OKK1tLOSIaJV0E3ANUAz+NiEWSLki33wBMAT4uaRuwGfg/6UzsvUi6tHNl/EVEzCpXWc3MzLJAOyY/d321tbXR0FDeU5rNzLoTSQ9HRG2ly2EJX9HLzMwsIxyUzczMMsJB2czMLCO61ZiypLXAc23MPgQo7yU9s8d17v56Wn3BdW6t/SJiaEcWxtquWwXl9pDU0NMmO7jO3V9Pqy+4zta1ufvazMwsIxyUzczMMsJBeYf6ShegAlzn7q+n1RdcZ+vCPKZsZmaWEW4pm5mZZYSDspmZWUb0+KAs6WRJT0paJunySpeno0jaV9IfJS2RtEjSxen6PSXdJ+np9HlwXp4r0vfhSUnvr1zp205StaRHJd2dLnfr+gJIGiTp15KWpn/v8d253pK+kH6mn5D0S0l9u2N9Jf1U0hpJT+Sta3U9JR0l6fF02//L3ZnPsqlHB2VJ1cD1wCnAWOBjksZWtlQdphH4YkQcAvwj8Nm0bpcDcyJiDDAnXSbdNhU4FDgZ+FH6/nQ1FwNL8pa7e30BfgDMioiDgcNI6t8t6y1pBPB5oDYi3klyB7qpdM/63kxS5nxtqeePgTpgTPoo3KdlSI8OysA4YFlEPBsRW4HbgdMqXKYOERGrIuKR9PUGki/qEST1+1ma7GfA6enr04DbI+KNiPgbsIzk/ekyJO0DfBC4MW91t60vgKQ9gBOAnwBExNaIeI3uXe8aYDdJNUA/YCXdsL4RcT/wSsHqVtVT0t7AHhExP5JZvbfk5bEM6ulBeQTwQt7yinRdtyJpFHAE8Bdgr4hYBUngBoalybrDe3EdcCnQlLeuO9cXYH9gLXBT2m1/o6T+dNN6R8SLwHeB54FVwOsRcS/dtL5FtLaeI9LXhesto3p6UC42ttKtzhGTtDswA7gkIta3lLTIui7zXkg6FVgTEQ+XmqXIui5T3zw1wJHAjyPiCGATaZdmM7p0vdMx1NOA0cDbgf6Szm4pS5F1Xaa+rdBcPXtK/buNnh6UVwD75i3vQ9IV1i1I6kUSkG+LiJnp6pfSLi3S5zXp+q7+XhwHfFjScpJhiPdK+jndt745K4AVEfGXdPnXJEG6u9Z7EvC3iFgbEduAmcCxdN/6FmptPVekrwvXW0b19KD8EDBG0mhJvUkmStxV4TJ1iHSG5U+AJRHx/bxNdwHnpK/PAX6Tt36qpD6SRpNMCPlrZ5W3vSLiiojYJyJGkfwd/xARZ9NN65sTEauBFyS9I111IrCY7lvv54F/lNQv/YyfSDJforvWt1Cr6pl2cW+Q9I/p+/XxvDyWRRHRox/AB4CngGeAr1S6PB1Yr3eTdFM9BixIHx8A3kYya/Pp9HnPvDxfSd+HJ4FTKl2HdtR9InB3+ron1PdwoCH9W98JDO7O9Qa+BiwFngBuBfp0x/oCvyQZN99G0uL9dFvqCdSm79UzwA9Jr+ToRzYfvsymmZlZRvT07mszM7PMcFA2MzPLCAdlMzOzjHBQNjMzywgHZTMzs4xwUDYrgaTtkhbkPXI3Apib3pVnoaQ/584XltRb0nWSnknv6POb9Nrcuf0Nl3R7un2xpN9JOkjSqPy7AqVpr5L0L51bYzOrhJpKF8Csi9gcEYc3s+2fIqJBUh3wHeDDwLeAAcBBEbFd0ieBmZKOSfPcAfwsIqYCSDoc2Iu3Xr/YzHoYB2WzjnM/cImkfsAngdERsR0gIm6S9CngvSQXddkWETfkMkbEAnjz5iFm1kM5KJuVZjdJC/KWvx0R/1WQ5kPA48CBwPOx8w1AGkjudwvQ0o0zDig41nCSOyOZWTfnoGxWmpa6r2+TtBlYDnwO2JPid+IRzd+5J98z+ceSdFUry2pmXZSDsln7/VNENOQWJL0C7CdpQERsyEt3JPA/6eszO7OAZtY1ePa1WQeLiE3Az4DvS6oGkPRxoB/wh/TRR9J5uTySjpY0oRLlNbPscFA2K81uBadETdtF+iuALcBTkp4GPgKcESngDOCk9JSoRcBV+D63Zj2e7xJlZmaWEW4pm5mZZYSDspmZWUY4KJuZmWWEg7KZmVlGOCibmZllhIOymZlZRjgom5mZZcT/B6lh3U1feWs1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(6,4))\n",
    "\n",
    "#plt.plot(accuracies)\n",
    "\n",
    "plt.plot( train_accuracy, '.b', label = 'training')\n",
    "plt.plot( test_accuracy, '.r', label = 'testing')\n",
    "\n",
    "axes = plt.gca()\n",
    "plt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n",
    "axes.set_xlabel('EPOCH')\n",
    "axes.set_ylabel('accuracy')\n",
    "plt.title('Un-normalized data accuracy', fontsize = 16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DATA NORMALIZED"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "norm = np.amax(training_data[0:500,:],axis=0)\n",
    "\n",
    "#normalize the data by the maximum value of each feature\n",
    "norm_training = np.zeros((500,15))\n",
    "norm_test = np.zeros((153,15))\n",
    "for j in range(15):\n",
    "    for i in range(500):\n",
    "        norm_training[i,j] = training_data[i,j]/norm[j]\n",
    "    for k in range(153):\n",
    "        norm_test[k,j] = test_data[k,j]/norm[j]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use same structure of ANN\n",
    "\n",
    "Train using normalized data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.512\n",
      "test accuracy 0.281046\n",
      "step 50, training accuracy 0.524\n",
      "step 100, training accuracy 0.718\n",
      "test accuracy 0.48366\n",
      "step 150, training accuracy 0.792\n",
      "step 200, training accuracy 0.824\n",
      "test accuracy 0.620915\n",
      "step 250, training accuracy 0.852\n",
      "step 300, training accuracy 0.862\n",
      "test accuracy 0.647059\n",
      "step 350, training accuracy 0.88\n",
      "step 400, training accuracy 0.884\n",
      "test accuracy 0.647059\n",
      "step 450, training accuracy 0.888\n",
      "step 500, training accuracy 0.894\n",
      "test accuracy 0.666667\n",
      "step 550, training accuracy 0.896\n",
      "step 600, training accuracy 0.902\n",
      "test accuracy 0.660131\n",
      "step 650, training accuracy 0.904\n",
      "step 700, training accuracy 0.904\n",
      "test accuracy 0.660131\n",
      "step 750, training accuracy 0.904\n",
      "step 800, training accuracy 0.908\n",
      "test accuracy 0.660131\n",
      "step 850, training accuracy 0.908\n",
      "step 900, training accuracy 0.908\n",
      "test accuracy 0.660131\n",
      "step 950, training accuracy 0.908\n",
      "step 1000, training accuracy 0.908\n",
      "test accuracy 0.666667\n"
     ]
    }
   ],
   "source": [
    "# Activation functions: https://www.tensorflow.org/api_docs/python/tf/nn\n",
    "\n",
    "#start an interactive session\n",
    "sess = tfc.InteractiveSession()\n",
    "\n",
    "\n",
    "#placeholders for input images and 1 hot class vector\n",
    "#inputs are 1x15 feature vector\n",
    "#note that the shape is optional, but will allow us to catch bugs later\n",
    "x = tfc.placeholder(tfc.float32, shape=[None, 15])\n",
    "y_ = tfc.placeholder(tfc.float32, shape=[None, 2])\n",
    "\n",
    "\n",
    "#define weights and biases, initialized as zeros for first hidden layer with 30 nodes\n",
    "#note that the default is to save a variable as tf.float32\n",
    "W_layer1 = weight_variable([15,30])\n",
    "b_layer1 = bias_variable([30])\n",
    "\n",
    "#assign the initial values to all variables\n",
    "sess.run(tfc.global_variables_initializer())\n",
    "\n",
    "#define the class predictions\n",
    "######## Activation relu \n",
    "h_layer1 = tfc.matmul(x,W_layer1) + b_layer1\n",
    "\n",
    "#define weights and biases, initialized as zeros for second layer \n",
    "#note that the default is to save a variable as tf.float32\n",
    "W_layer2 = weight_variable([30,30])\n",
    "b_layer2 = bias_variable([30])\n",
    "\n",
    "#define the class predictions\n",
    "######## Activation tanh\n",
    "h_layer2 = tfc.nn.tanh(tfc.matmul(h_layer1,W_layer2) + b_layer2)\n",
    "\n",
    "#define weights and biases, initialized as zeros for second layer \n",
    "#note that the default is to save a variable as tf.float32\n",
    "W_layer3 = weight_variable([30,2])\n",
    "b_layer3 = bias_variable([2])\n",
    "\n",
    "#define the class predictions\n",
    "######## Activation softmax\n",
    "h_layer3 = tfc.nn.softmax(tfc.matmul(h_layer2,W_layer3) + b_layer3)\n",
    "\n",
    "\n",
    "#define the cross entropy loss\n",
    "#the cross entropy logits function applies a softmax on the model's unnormalized prediction and sums across all classes\n",
    "#the reduce_mean function takes the average over the sums\n",
    "cross_entropy = tfc.reduce_mean(tfc.nn.softmax_cross_entropy_with_logits_v2(labels=y_, logits=h_layer3))\n",
    "\n",
    "#define a training step, which you will run later with the train_step.run command\n",
    "#train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)\n",
    "train_step = tfc.train.GradientDescentOptimizer(0.1).minimize(cross_entropy)\n",
    "\n",
    "#define a true/false vector to evaluate predictions\n",
    "correct_prediction = tfc.equal(tf.argmax(h_layer3,1), tf.argmax(y_,1))\n",
    "\n",
    "#recast the true/false vector as a binary vector and take the average\n",
    "accuracy = tfc.reduce_mean(tfc.cast(correct_prediction, tf.float32))\n",
    "\n",
    "#add a tensorboard graph\n",
    "writer = tfc.summary.FileWriter('./norm_nn_graph', sess.graph)\n",
    "# add a summary to store the accuracies\n",
    "training_summary=tfc.summary.scalar('train_acc', accuracy)\n",
    "testing_summary=tfc.summary.scalar('test_acc', accuracy)\n",
    "\n",
    "\n",
    "#initialize all variables\n",
    "sess.run(tfc.global_variables_initializer())\n",
    "np.random.seed(1)\n",
    "\n",
    "train_accuracy = np.zeros(1001)\n",
    "test_accuracy = np.zeros(1001)\n",
    "\n",
    "#run 1000 iterations of SGD with batch size equal to 500\n",
    "for i in range(1001):\n",
    "    batch = next_batch(500, norm_training, training_label)\n",
    "    \n",
    "    train_accuracy[i],summary = sess.run([accuracy,training_summary],feed_dict={x:batch[0], y_: batch[1]})\n",
    "    writer.add_summary(summary, i)\n",
    "    \n",
    "    test_accuracy[i],summary = sess.run([accuracy,testing_summary],feed_dict={x: norm_test, y_: test_label})\n",
    "    writer.add_summary(summary, i)\n",
    "    \n",
    "    if i%50 == 0: #evaluate and print the training accuracy every 10 iterations\n",
    "        print(\"step %d, training accuracy %g\"%(i, train_accuracy[i]))\n",
    "    if i%100 == 0: #evaluate and print the testing accuracy every 100 iterations\n",
    "        print(\"test accuracy %g\"%accuracy.eval(feed_dict={x: norm_test, y_: test_label}))\n",
    "\n",
    "    train_step.run(feed_dict={x: batch[0], y_: batch[1]})\n",
    "    \n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Normalized data accuracy')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAEYCAYAAAAd7fxUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2XElEQVR4nO3de3hU1b3/8fc3NyDchSiKKChYxXoFsbFVqNiqrYqK7aHK8VIVqLWtnloCtedXe1NAre2pN3KsWizW01OVeqhVCy1I26kaFazcFBURgQoqchGBJOv3x9pjdiaTZJLMnkvm83qeeWb22mvvWXsyyTfrstcy5xwiIiKSOUXZLoCIiEihUfAVERHJMAVfERGRDFPwFRERyTAFXxERkQxT8BUREckwBd88YmaXmpkzs61m1jdhX0mw74YsFa/dzOx+M1sb2h4cXMul2SpDG48dE5R3TDuOvcHMTm3P+4pI/lLwzU+9gapsFyJCG4FK4A/ZLkgGfB9Q8BUpMAq++ekp4BtmNiCqNzCzLlGduzXOud3OuX845zZnqwwSPTMrNrOSbJdDJBsUfPPTj4Pn61vLaGajzGyBme0ws51mttDMRiXkud/M1ptZpZn93cx2AbNCzb9TzOwmM9tkZtvN7NdmVm5mQ83syeDca8zskoTzDjWzB8zsDTPbZWavm9ldiU3mScrcqNk51Nye7HFD6Lj+wfnfNrPdZrbKzCYlOf9YM3vBzD4ys9fMbHJrn2Po2Aoze9DMtgXN/3OAPknyfd7MHjezjWb2oZm9bGbfNrPiUJ749HLXJ16PmZ1gZr8Lfi67zGy1md1oZt1SKGPKx5rZeWb2t+BnuM3MnjWzc0L7S8ysysxWBJ/XZjN7wswOD/bHfzaDE857Q+j6Pr5eM/uJmU0zszeAPcBRZtbVzG4LPqMdwffs/+LvkXCOIcF3alPwM37dzH4e7LsuSKtIOMaCfL9p7bMTyRT915mfNgK3A9eY2S3OuTeTZTKzo4HFwArgUsAB04DFZvYp59yyUPbewEPALcB3gV2hfdOBRcAlwHBgFlAPHAf8d3DM14D7zKzGObc8OO4AYD1wDfA+cEhw7sfxzcqp+kOS/BcBVwMrg2vtBfwN6AbcALwBnA7cZWZdnHO/CPIdEbx/DTAB6BLk7wHUpVCWR4Bjgut4Ffg34BdJ8h0CLAz2fQSMDN6nAv8zILimGHA/MDtIWx88HwQsDfZtB44E/l9w3gmtlDGlY83sG8B/AfPwP9sdwPHA4NC5HgLOBX4GLAC6AqcA+wOrWilHMpcCrwPXATuBDfifQU/8P5UbgX2Aq4B/mNnhzrlNQXmHAM8CH+Kb618FBgGfD859L/Aj4DL8dzTu88AQ4KvtKK9INJxzeuTJg4YAOhT/B2orcG+wryTYd0Mo/++CPH1Cab2A94BHQmn3B8eOS3i/wUH6nxPSHwnSJ4bS+gK1wPdbKH8J8Jng2OMS3n9tkve9tJnzfBof0H4aSvvPIG1YQt7/BrYAJcH23GC7eyjPIHwtbG1zZQ/yfS4o14SE9D8G6WOaOc6Ca78e/09IUWifA37cyvvGj5+I/6enXxu+M0mPDb4H28PfgyTHnhqU75spfCcHJ6Tf4P+8NEpz+GDbrZUyFwPlQfmuDaXPwf+DcEALx94PrAEs4fu6Kh2/g3roka6Hmp3zlHPuPeBW4GIz+0Qz2U4B5jvntoaO2wY8BoxOyFsLzG/mPH9M2I7XeJ4Mnfd94B18IAPAzMrM7LtB8+8uYC+wJNjdXJlbFDRvPhq893WhXWcAzwBvBE2lJeb7E58E+uFr7OBrm48753aGyv4Wvtbcmkp87fjhhPSHkpRzfzObbWZv4gP7XnzNrg+wb2tvZGa9zGymmb0G7A6OfwAfTIel4diT8LX96hZO9Xl8wPzv1srbBk8453YlJprZl83sGTPbiv8u7gzKF/6efB7/fd7QwvnvBA4Fxgbn3R84m4aWBZGcoOCb327D12J/2Mz+ffDNeIk24WuqYe8455prdn0/YXtPC+ldQ9s34WtAvwa+CIwCzg/2daWNgqbl+fim2Qudc/Wh3fvi/9nYm/D432B/v+B5f+BfSU6fLC3R/sD7zrm9LR1rZkX4f3DOwgfcU4ETgJ8EWVK59vuAKfhm4c8Fx389xeNTOTb+eaynef2A95IFyw5o8n00s7OB/8F3IVwInIgv82YaX2s/Wi4vzrln8V0KU4KkK/DB/FcdLbhIOqnPN48553aY2U34GvDNSbK8ByQbET0g2NfodGkuHvj+xTnOufgAMcysR3tOFAxUegj/T8OocM018C6+5v2tZk6xOnjeCOyXZH+ytEQbgb5mVpoQgBOPPRTfx/vvzrlfh67h7BTeAzPrCozDdyH8PJR+VBqP3RI8DwRebuZ0W4B9zKxbCwH4o+C5LCG9X2LGQLLv2QRgjXPu0lB5S/H/PCaWZ2Az5w27C5htZgPxwfd/g5YikZyhmm/+uxN4m4YR0GGLgS+aWc94QvD67GBf1Mrxtc+wy9p5rp/ia7ZnOefeTrL/CeBwYJ1zribJY3uQLwZ8wcy6xw80s0H4fuTWxPD9keMT0hMHQJUHzx9fexBMLkpyzj34QWJhXYL3SfzsLk2hjKke+3d8/2mT0eAhT+Gbqq9oIU98sN8n4wlBc//nk2dPqhxfOw37d/x1JJbnrKApuSW/wfcXP4gffHZ3G8oikhGq+eY559xuM/shyfvufoRv+lxoZjPxtY4q/B+75pqq0+kJ4BIz+yd+EMz5+L7GNjGzCcA38c3YXczsU6Hd651z6/FN8P8GLDGz2/A13e74gHyyc25ckP/HwJeAp8zsZnyN7Qek0OzsnPuTmf0VX6vqT8No508mZF2JD0o/MbM6fCC8tpnTrsD/g/QEvhl/g3Nug5n9A/i2mW3E1/i+Sgq1PufcB6kc65zbbmbTgV+Y2cP4gWjbgWOBj5xzv3DO/SXY99PgH5Q/A6X4f4L+4JxbBDwHvAbcHDS378aPVG7LfeJPAOcGP7f5wAj8z3trQr7v47sv/m5mN+K/UwOBM5xzE0PXtsvM7sd/5v90zv29DWURyYxsj/jSI/UHodHOCeklwCskjHYO9p2Iv0VkB34Qy0J8s204z/34IJb4foODc16RkH5DkF6SkL4W+HVouz++qfj94DEX35fXaCQzrYx2Dr1fsscNoeP64oNw/B7Sd/ADvK5JKOdpwIv4QPE6MDmxDC38DCpoqFltxY/AHUfCaGd8EPsr/raY9fh/dq4gYWQwvsb9PL759uPrCT6DPwbv8w7+1rIvJr5PM2VM+VjgAvxAtV3AtuD1WQnfrevx3689+H7Yx4FPhPIcib8VbQewDvgPmh/t3GRkN74F7sf4kdAf4ltljsN/n+5PyHto8PlvCf38bktyzsrg/b6e7d9bPfRI9jDnoujqExHJHjP7Cb7//wDnR/iL5BQ1O4tIp2Fmx+FvT/oWUK3AK7lKNV8R6TTMr0y1H/7+7n93DQPtRHKKgq+IiEiG6VYjERGRDMu7Pt/+/fu7wYMHZ7sYIiJ55fnnn9/inKtoPadkQt4F38GDB1NTU5PtYoiI5JVgnnHJEWp2FhERyTAFXxERkQxT8BUREcmwvOvzFRGR7Hn++ef3LSkpuQc/p7kqcMnVAy/X1tZeMWLEiHeSZVDwFRGRlJWUlNwzYMCAIyoqKt4vKirSRBFJ1NfX2+bNm4dv2rTpHuCcZHn0X4uIiLTFJysqKrYp8DavqKjIVVRUfEDTFc8+ppqviGRdVRXMng07d4JzYAZFRVBf37ANhb2vqAiOOALuugsqK6P7WaSgSIG3dcFn1GwFV8FXpBOqqoI77oCPPsrNQBLet3ev35bWLVsGJ58MS5ZkPQBLB0Xa7GxmZ5jZajNbY2bTkuzva2aPmtlLZvasmTVbRRcRr6oK+vSBLl2gtBRKSvxz/HVREcya5WuRdXVQW+sf8dd79jTezvY+Bd62qauDRYuyXYrs2rJlS/GMGTPaPFvX6NGjh27ZsqW4pTzXXHPNAfPmzevZ/tKlJrKar5kVA3cAn8MvJv6cmT3mnFsRyvZdYKlz7jwzOzzIPzaqMonku+HDYeXKbJdCsqm4GMaMyXYpsuvdd98t/uUvf7nvtGnTNofTa2trKSlpPqwtXrx4TWvn/tnPfrYhDUVsVZQ131HAGufc6865PcBDwLiEPMOBhQDOuVXAYDPbL8IyiURq4kTo2rVxTbS0tPlaalv2mRVG4C0u9tdbVuaf49vh14W4r6wMjjkmP5ucFyyg+/TpDFiwgO7pON+3v/3tA996660uhx9++PBPfvKTR5x44omHnX322UM+8YlPHAlw2mmnHXrkkUceMXTo0CNvueWW/vHjBg4ceNTGjRtLVq9eXXbIIYccOWHChIOHDh165Kc//elhO3bsMIDx48cPvu+++/rG81977bUHDB8+/IjDDjts+IsvvtgVYMOGDSUnnXTSsOHDhx9x4YUXHnzAAQcctXHjxjZVZqMMvgOBt0Lb64O0sGXA+QBmNgo4GDgw8URmNsnMasysZvPmzYm7RSJVXQ2DB0P37i0HyqIimDsXdu+Opsm2vXIxkCTu69oVLrrIN0HX1vp+4N27/XN8O/y6EPft3g1Ll+Zn4D3rLA6bNYuBZ53FYekIwLfeeuv6QYMG7V61atWKGTNmrH/ppZe633zzzW+/9tprywHmzp27dvny5SuXLl26Yvbs2ftt2rSpSVPzunXrun7zm998Z82aNct79+5dN2fOnL7J3qt///61K1asWPnVr35184wZM/YDmDZt2gGjR4/evmLFipXnn3/++xs3bixr6zVEOeDKkqQl9u7MAH5uZkuBfwIvArVNDnKuGqgGGDlypHqIpE1iMZg2DZ57zgez1gYBde0KQ4f6tBUr/DG5yMwHsWTX0KMHTJoEM2dmt4wiCxfSc+9eiurrobaWooUL6XnaaexM53scffTROw8//PCPf1Nnzpy53x/+8Ic+AJs2bSpdvnx51wEDBjR6z4EDB+4+6aSTdgEcd9xxH65du7ZLsnNfeOGF7wOMGjXqw8cee6wvwLPPPttj3rx5awAuuOCCbb169Wrzv8dRBt/1wKDQ9oFAo7Z059w24DIAMzPgjeAh0m7V1XDjjbB5s68ttLXWuGOHr2HksiOO8P8YiOS6sWPZfttt1NfWUlRSQv3YsWxP93uUl5fXx1/Pnz+/5+LFi3vW1NSs6tmzZ/2oUaM+sWvXriatvGVlZR9X5IqLi12yPABdu3Z1ACUlJa62ttYAXBpGCUbZ7PwcMMzMhphZGTABeCycwcz6BPsArgCeDgKyFLhUm3oT9xUXw+TJ8Oab8OGHHWuu7QizaJps+/SBqVMVeCV/nHYaO+fP55XvfIe358/nlXTUenv37l23c+fOpPFr69atxb17967r2bNn/Ysvvth12bJlaelnDhs1atSOBx54YB+ARx55pNe2bdtaHEGdTGQ1X+dcrZldDTwJFAP3OueWm9mUYP/dwBHAHDOrA1YAl0dVHskfp58OTz2V7VIkV1zsA2tz96yWl6u5VyTRaaexM51NzQMGDKgbMWLEjmHDhh3ZpUuX+oqKir3xfePHj/+gurq64rDDDht+6KGHfnTMMcektYkbYMaMGRsuuOCCQ4YPH963srJyR0VFxd4+ffq06V99S0f1OZNGjhzpampqsl0MSbNYDK66yk8iEOVXMt5P2lyfbzwt8ZiDD4bp031gFclHZva8c25kR8+zbNmytcccc8yWdJQpX+3atctKSkpcaWkpCxYs6H711VcfvGrVqibtUcuWLet/zDHHDE52Ds1wJVlXVeUnhYhCvKbavz/84AetB89YzM8gFG6uvvtuBV0RabBmzZqyL3/5y4fW19dTWlrqZs+evbat51DwlayIT3/44Yet13Rba+pN5xy4lZX+PspZs2DDBrj8cgVeEWnsqKOO2r1y5coOjbxQ8JWMCE+cX9vkZrLkevWCm2/OfPCrrIRHH83se4pIYdGSgtJmsRgcd5wfiRsffVxa6kcmd+8OPXvC6NE+X1WVH5E8axZ88EFqgbdLFz+i94MPVOsUkc5JNV9JWXxQVHP3wIYD69NPw0kntf09pk7VSGER6fwUfKVF8ebiHTuiu2e2Wzc44QSYMSP/ps4TEWkPNTtLUrEYDBrU0FycrsAbv9WnRw9fy3XOD7pavFiBV0RS094lBQF++MMf7rt9+/aPY18qywxGQcFXPhaL+b7a8nLfZLx+fevHFBX5YJqKiy7yI5Nra2H7djUvi0j7xJcUbM+xs2fP3m/Hjh0fx77Fixev6d+/f8bnwlOzswA+8H7mMz44pmKffeCmmxoGRMVifoHvMWNg3jzfVL1rlx+UdfzxalIWKWgLFnRn4cKejB27ndNO6/CMU+ElBUePHr1t33333fvoo4/us2fPHvviF7+49bbbbtuwbdu2onPOOeeQjRs3ltXX19vUqVM3/Otf/yp95513SkePHn1Y3759a5955plXBg4ceFRNTc3Kbdu2FZ155pnDRo0ataOmpqbHfvvtt+fJJ59c06NHD7d48eLyK6+8cnB5eXn9iSeeuOPPf/5z71dffXV5R65BNd8CF59D+eSTWw+8JSUwYIAPrO++23gkcmWlnwGqstLXaLdu9YsabN+uJmWRgrZgQXfOOuswZs0ayFlnHcaCBWldUvD000/ftmbNmq4vvfTSypUrV65YunRp+R//+McejzzySK8BAwbsXb169YpXX311+fnnn7/te9/73jv77rvv3sWLF7/yzDPPvJJ43uaWGbziiiuG3HHHHW8uXbp0VXFxcVrm4FPwLWDV1Q2LELTUp2sGf/+7X1N040bd/iMibbBwYU/27i0K+pyKWLiwZzpP/8QTT/R6+umnew0fPnz4kUceOfy1117rumrVqq7HH3/8riVLlvT62te+NvCJJ57o0a9fv1ablpMtM7hly5binTt3Fn3uc5/bCXDJJZe8l45yK/gWqKoq+NrXmt9v5mu5554Lf/ubaq4i0k5jx26ntLQ+WJqrnrFj07qkoHOOa665ZuOqVatWrFq1asW6detevvbaa7ccffTRu1944YUVRx111K7rr79+4HXXXbd/a+dKXGawtrY2svUPFHwL0MSJfhRzS83Md9/ta7mPPqrAKyIdcNppO5k//xW+8523mT//lXT0+YaXFDzzzDO3PfDAA/0/+OCDIoA33nij9O233y5Zu3Ztac+ePeuvuuqq96655pp/LV26tByge/fudfG8qaioqKjr3r17/cKFC7sDxJcS7CgNuCowsRg8+GDyfcXF/vYird4jIml12mk70xF048JLCp566qkffOlLX3rvhBNOOBygvLy8fu7cuW+sWrWqy/Tp0w8sKiqipKTE3XnnnW8CXHLJJVvOPPPMYfvuu+/eZP2+ycyePXvtlClTDi4vL6//9Kc/vb1nz54dHh2tJQULSHU1fPvbfsKMRMXFfkEB1XJFOictKdh+H3zwQVHv3r3rAb773e8O2LhxY+l99933VmvHaUnBAhVezKC+Pnkzc2mpD7i6FUhEJLnf/va3vW+99db96+rqbODAgbsffPDBtR09p4JvJzVxIsyd23q+z34Wnnwy+vKIiOSrK6+88v0rr7zy/XSeUwOuOplYDA47LLXACzB+fLTlEZFOp76+vt6yXYhcF3xGzQ5rVfDtROKzVL36aut599nHN0lrYJWItNHLmzdv7q0A3Lz6+nrbvHlzb+Dl5vKo2bkTWbSo9VmqiorgrrsUdEWkfWpra6/YtGnTPZs2bfokqsA1px54uba29ormMij4diJbtyZPnz0bjjqqYe5lDawSkfYaMWLEO8A52S5HvlPw7SRiMbjllsZp5eVw220NtVwFXRGR3BBpk4GZnWFmq81sjZlNS7K/t5n9n5ktM7PlZnZZlOXpzObMadrkfPHFal4WEclFkdV8zawYuAP4HLAeeM7MHnPOrQhl+zqwwjl3tplVAKvNbK5zbk9U5eoswkv4/fOf8KtfNd5fVOSDr4iI5J4om51HAWucc68DmNlDwDggHHwd0NPMDOgBvAfURlimvFZdDTfeCJs2+eX6WnLddWpmFhHJVVEG34FAePqt9cCJCXluBx4DNgA9gX9zzjUZr2tmk4BJAAcddFAkhc1FsRhMmwYvvOAXpm9p2b9EffpEViwREemgKPt8k90DljiR9OnAUuAA4FjgdjPr1eQg56qdcyOdcyMrKirSXc6cU1UFFRVw0knw9NN+Lua2BN7iYt8cLSIiuSnK4LseGBTaPhBfww27DHjEeWuAN4DDIyxTzosv97elndOWDx2qBRJERHJdlMH3OWCYmQ0xszJgAr6JOWwdMBbAzPYDPgG8HmGZcloslvq0kJakXWHqVD+7lQKviEhuiyz4OudqgauBJ4GVwG+dc8vNbIqZTQmy/Qg4ycz+CSwEqpxzBbVUVdicOS3v79YNBg/2k2bU1/vnI46A4cP965kzM1JMERHpIK3nm0NGj/Z9vMloHmYR6Yh0recr6aEZrnJELAZ//WvjtJ49YdAg+Na3FHhFRDoTBd8ckWxRhIsu8osgiIhI56IVKXLE8uWNt4uLNUOViEhnpeCbAyZObDrKecQIjVoWEemsFHyzLFngBbj88syXRUREMkPBN4uau6/32GM1wEpEpDNT8M2i5u7r/dSnMlsOERHJLAXfLFqwoGlaWZkGWomIdHYKvlkycSKsWdM47cAD/S1HGmglItK5KfhmQXV18r7e//xPBV4RkUKg4JsFP/950zQNshIRKRwKvhkWi8HKlU3T77wz82UREZHsUPDNsDlzIHEti9mz1dwsIlJIFHwzbMWKxtunnKLmZhGRQqPgm0GxGCxZ0jhtn32yUxYREckeBd8MmjWraZPzgAHZKYuIiGSPgm+GxGIwb17jNDNNqCEiUogUfDPkqquapo0bp4FWIiKFSME3A04/HZYubZxmBlOnZqU4IiKSZQq+Eauqgqeeapr+ne+o1isiUqgUfCP24INN0449FmbOzHhRREQkRyj4Rqx//6Zpms1KRKSwRRp8zewMM1ttZmvMbFqS/d8xs6XB42UzqzOzTnPnaywGy5c3bJtpNisREYkw+JpZMXAHcCYwHPiKmQ0P53HO3eycO9Y5dywwHVjsnHsvqjJlUiwGF14Ie/c2pI0bp9msREQk2prvKGCNc+5159we4CFgXAv5vwL8JsLyZERVlV+X96STYO3abJdGRERyUZTBdyDwVmh7fZDWhJmVA2cADzezf5KZ1ZhZzebNm9Ne0HSpqvKzWL39dvL9ms1KREQg2uBrSdJckjSAs4G/Ndfk7Jyrds6NdM6NrKioSFsB0y3ZyOY4zWYlIiJxUQbf9cCg0PaBwIZm8k4gz5ucYzHYuLH5/XffrYFWIiLiRRl8nwOGmdkQMyvDB9jHEjOZWW9gNPD7CMsSuUWLkqfHRzhroJWIiMSVRHVi51ytmV0NPAkUA/c655ab2ZRg/91B1vOAp5xzO6MqSyaMGQNFRVBX5wNu//5wxBEwY4ZqvCIi0lhkwRfAOfc48HhC2t0J2/cD90dZjkyYN6/htiLn4Mc/Vm1XRESS0wxXaRCLwS23NE57OOm4bREREQXftFi0yNd2w8aPz0pRREQkDyj4psGYMVBW5l/HlwpUk7OIiDRHwTdNnPOBt7QUzj0326WRDquqgj59oEsX/0MtKYGePX36xInQtatPj+8rLYXychgyBKqrs116yRVVVdCjR8N3JPx9CX+3UtnXpYtfEi0Wy/ZVSRpEOuCqUCxa5Ec5O+efFy3SCOe8Fp+qLNGOHcnT42pr/Zyikyf7bTV/FLbmvkcdsWwZnHwyLFmiPzJ5TsE3DcaM8f+g1tf75zFjsl2iHFRV5W943rnTf0hHHw3r1sGWLX5/UZH/AMHfp/WDH2QmeMViMG0aPPcc7Nnjmy9qazt+3smT4etfb2gSiV9ffBu0r7Pvq6tr//enJfoPv1NQ8E2T+ICrxIFXQtMaQG0tPPts8/k3bcpM7TEW87WIqP5IpiOIiyQqLtZ/+J2A+nzTYNEi/3fWOf/c3GxXBaW6GgYP9n2j7W16mzy5+f6v0lLYf//m+1fD799cn9pnPpNa4C1q4dekpMT/MWwpjwj470r8+1JS4kdphrdb21dWBsccoybnTiKlvxhm9rCZfdHM9Bcmia1bG1pM6+uhX7+sFif7qqt94HzzTdi9u2Pnqq31zcG1tT5Q1tY2POI15MQAnPj+4ePir/fsafihtaSkBP76Vz+EPdHUqX5mldpan0cBWJoT/67Evy979/rvZni7tX27d8PSpQq8nUSqzc53AZcB/2Vm/wvc75xbFV2x8kcsBj/9acO2Gbz7bvbKk3WxGFx3XduOKSryQa62NrWAmGjyZLjqqob+to4095aU+PKUlcHxxzfMDxr/gzd7tn+fSZNg5syG4yorfQC+6ipYvrxxP0Qu9UNqX2b39ejR9LsiQorB1zm3AFgQLILwFeBPZvYW8N/Ar51zeyMsY05btKhxvCjoAVexmG/KbSmAjhrVuL+3pASeftoHr470waaj37a1FTBmzmz5j2hlJbz4YsfLISKdXsrtZGbWD7gUuAJ4Efg5cDzwp0hKlifGjPHdh/HK2+23F1CrUGK/anOBN14DmDoVnnnGPw8cCKec0hB4wT8vWeLTu3Vrvv+rrc27Zs33qZWUwIABWnpKRDLKXArDc83sEeBw4AF8k/PG0L4a59zI6IrY2MiRI11NTU2m3i4l1dV+Lufx4wvo73e8X7U1xcXpHyDSlhpyuGYtUsDM7PlM/q2WlqUafE91zv05A+VpVa4F31gMxo7143fKymDhwgL5O7///n7AU0t69ICnnormA4nfn/vCC/DRR03728rK4IQTtKajSEDBN7ekOuDqCDN7wTm3FcDM+gJfcc7dGVnJ8sSiRX4QYn29fy6Ie98nTmw98ALcemt0H0ZlJSxeHM25RUQilmrn2ZXxwAvgnHsfuDKSEuWZfv3y/Daj8DzFXbpA375+UoyWJFsvMd6v2q2b7wdWH6qISLNSrfkWmZm5oI3azIqBsuiKlT/efbfhToOiojy7zWjiRJg7t3Hanj0Nk2IkG9lbVeWbecOi6NcVEenEUg2+TwK/NbO7AQdMAZ6IrFR5JD7aOd7nmzO3GVVXw403wq5dcOmlyQPpvHnNH3/rrXDoof71jTf6Zub4JBVhpaW++VeBV0QkZakG3ypgMvA1wICngHuiKlQ+qaz0g6wWLfKBNydiUOJI5GQ12epqv8hBc+rqUhvNfO21OXLRIiL5I6XRzrkk10Y755R4bXfduqYrPAwdCuefD3fc4ZuN4zPydETv3n5uTRHJeRrtnFtSndt5mJn9zsxWmNnr8UfUhcsHsRjcdFMOrG8dns84WVDdZx9fA965s2Hx4bCLLvJ9t22RSs1YRESaSLXZ+T7g+8BtwGfx8zxbVIXKFzl1j2+yEchxRUUtL+EHcOSRftDUhRf6BeGTMfMBWvPVioh0SKq3GnVzzi3EN1O/6Zy7ATi1tYPM7AwzW21ma8xsWjN5xpjZUjNbbmZ5dePmokU+8NbV+eesLiV47LHN72ttsYL4hNSVlfDgg8lrwMXF8Le/+dVV3n9fgVdEpANSDb4fBcsJvmpmV5vZecC+LR0Q3I50B3AmMBz4ipkNT8jTB7gTOMc5dyTwpTaWP6vGjPFxK36La9ZGOsdi8MADjdOKi311vDklJb4G29r8ylpDVEQk7VJtdr4GKAe+CfwI3/R8SSvHjALWOOdeBzCzh4BxwIpQnguBR5xz6wCcc++kXPIcEV45LiuSrSQUv+923rzkC9lPndr66jyaPUpEJDKtBt+gBvtl59x3gB34/t5UDATeCm2vB05MyHMYUGpmi4CewM+dc3OSlGESMAngoIMOSvHto7doUcPYpbq6LE0tmbimIUB5eeM1aOMjnLt18+vNqslYRCSrWg2+zrk6MxsRnuEqRckGZCUeXwKMAMYC3YCYmf3DOfdKQhmqgWrwtxq1oQyRGjPGt8pmdYKNZPNZnnNOw+vW1qAVEZGMS7XZ+UXg92b2v8DHMzM45x5p4Zj1wKDQ9oHAhiR5tjjndgI7zexp4BjgFfJATkywEZ7fEvzAq1//OgsFERGRVKUafPcB3qXxCGcHtBR8nwOGmdkQ4G1gAr6PN+z3wO1mVoKfK/pE/O1MeSPcupsVifNb3lnwC02JiOS8lIKvcy7Vft7wMbVmdjV+Xuhi4F7n3HIzmxLsv9s5t9LMngBeAuqBe5xzL7f1vbIlFoM5QQ/1xRdnIQjH17Tt0gX22w+mT9eIZBGRPJDS9JJmdh9N+2txzn01ikK1JFeml4zF4OSTG9YZ6NIF/vKXDMa+xAKAv30ofNuQiEhA00vmllTv850P/CF4LAR64Uc+F6xZsxrHvd27MzzJRnyodVhtbZZn+hARkVSk2uzcaO5CM/sNsCCSEuWJ1asbb5tleLRzslHOWZ3pQ0REUpVqzTfRMCB3brjNgoqKxtsnn5zh1t533/URP27wYDU5i4jkiZRqvma2ncZ9vpvwa/wWpFgM/vGPhu3iYpgxI8OF6Nev8bRaGmwlIpI3Um127hl1QfJJuLvVDK68MgtxL3x/b1GR3xYRkbyQ6nq+55lZ79B2HzM7N7JS5bj4zFbFxdC1q7/NKCuF6NLFF6JLF/X1iojkkVQn2fi+c+7R+IZzbquZfR+YF0mpclxOzGwFcEmwtkVWbjIWEZH2SjX4Jqshp3qspFssBmPHNsxqlZWqt4iItFeqo51rzOynZnaomR1iZrcBz0dZsFxWXQ2jR8P3vudjYCwW8RvGYv4Ny8v97USf+Qzs2uU7nj/6SPf2iojkmVSD7zeAPcD/AL8FdgFfj6pQuSwWg699Dfbu9WOdIp9cIz6T1dNPNwTc8BKCzsHy5REWQERE0i3V0c47gWkRlyUvTJvWOPbV1yef7yJtks1kleiZZyIsgIiIpFuqo53/ZGZ9Qtt9zezJyEqVo2IxWLKkaXqkd/mMGeNHNLfk/PMjLICIiKRbqs3O/Z1zW+Mbzrn3gX0jKVEOW7So8bwWAKWlabzLp7oaTj/dPwNMnAif/ay/mbiszPf3Fhf757Iy6NMHpk6FmTPTVAAREcmEVEcs15vZQc65dQBmNpgkqxx1dlu3Nt4eOtQvKZiWu3yqq2HyZP/6qafgl7+EZ59tmm/2bJg0KQ1vKCIi2ZJqzfd64K9m9oCZPQAsBqZHV6zclDiw6pBD0hh4v/WtxmnJAi/Aww8nTxcRkbyR6oCrJ8xsJDAJWAr8Hj/iudOrqoK5c33L7/r1jfeNH5+GNwjXeFORljcVEZFsSnVhhSuAbwEH4oPvp4AYcGpkJcsBVVV+3d5khg1LU+vvz3+eWr7Bg/3iCWpyFhHJe6k2O38LOAF40zn3WeA4YHNkpcoRDz7Y/L4PP0zDG1RXw4oVreebOhXeeEOBV0Skk0h1wNVHzrmPzAwz6+KcW2Vmn4i0ZFk2cWLTZuawiy5Kw5skq/WWl/sZPJyDHj18wNVoZhGRTiXV4Ls+uM93HvAnM3sf2BBVoaJSVeUHC+/c6WObWcOqfPFtgNrals9z0UVpiIfJar3FxbBggRZJEBHp5FIdcHVe8PIGM/sL0Bt4IrJSRWDiRD9wqqPSdlttslrviBEKvCIiBaDNKxM55xZHUZAoxWIt99+mIq3jnZrr67388jScXEREcl2qA67axczOMLPVZrbGzJrMDW1mY8zsAzNbGjz+XxTlSDYzVap69fJN1Wkd75TsXt1zz9WAKhGRAhHZmrxmVgzcAXwOWA88Z2aPOecSq3xLnHNnRVUO8NM/duvmFwWChqmSm+vzdc7nv+qqiMY6HXusn8UqrrTUt2eLiEhBiCz4AqOANc651wHM7CFgHJDCvTXpVVkJCxf6GvCYMVnuVo3F4Gc/a9g2g9tvV1+viEgBiTL4DgTeCm2vB05Mkq/SzJbhR09f55yLZHHaysoMxbdYLHmUj6evW+dvJQqLdFkkERHJNVEGX0uSltjz+gJwsHNuh5l9AX8r07AmJzKbhJ/akoMOOijNxUyjWAzGjoU9e/yqQwsX+gAcTo+vShQPwGVlaVwWSURE8kGUwXc9MCi0fSAJ9wY757aFXj9uZneaWX/n3JaEfNVANcDIkSNzdzWlRYtg927fibxrF1xyie9A7t69ocO5rs73+ZaVwQEH+L5eNTmLiBSUKIPvc8AwMxsCvA1MAC4MZzCzAcC/nHPOzEbhR1/nbxtsv34+8Ma9+mryfEuX+r7erl010EpEpABFdquRc64WuBp4ElgJ/NY5t9zMppjZlCDbBcDLQZ/vfwETnGvvTUE5oC19t875WnLiOoUiItLpRVnzxTn3OPB4Qtrdode3A7dHWYaMGjPG9+nW1aWWv74etm6NskQiIpKDIp1ko+BUVsLZZ7ftmKVLIymKiIjkLgXfdIrF4PHHW88XNn58NGUREZGcpeCbTnPmNL2HN9HBB0OfPn6y6NmzNaWkiEgBirTPt6DEYnDvvS1PIl1aCr/5jW4tEhEpcKr5psuiRQ0LAZv5hRI+//mGCaPN/KpFCrwiIgVPNd90Cd/j6xyceSYcdRQsWdIw49XFF2e3jCIikhMUfNMhFvPLBJr5wFtU5O/5zakVHUREJFco+HZUfN7m3bsbAm+XLg3zNWdsRQcREckX6vPtqPB8zgCHHOKXDFTAFRGRZij4dlTifM6vvQbXXONrxCIiIkko+HbUiy823nbOD7DSnM0iItIMBd+OiMXgnnuaphcVaY1eERFploJvR4Tv7Q077jj1+YqISLMUfDuiuRWJhg3LaDFERCS/KPh2xPz5ydM3b85sOUREJK8o+LZXdTWsWJF8n1YqEhGRFmiSjfZ6+OHG2wMGwNFH+8CrlYpERKQFCr7tNX48PPVUw/YPfqCgKyIiKVHwba9Jk/yEGo88Auefr8ArIiIpU59ve8Vi8ItfwBtv+GfNaCUiIilS8G2vRYv8TFZ1dZrRSkRE2kTBt73GjPFr9BYX+2fNaCUiIilSn297aa1eERFpp0hrvmZ2hpmtNrM1ZjathXwnmFmdmV0QZXnSrrISpk9X4BURkTaJLPiaWTFwB3AmMBz4ipkNbybfTODJqMoiIiKSS6Ks+Y4C1jjnXnfO7QEeAsYlyfcN4GHgnQjLIiIikjOiDL4DgbdC2+uDtI+Z2UDgPODulk5kZpPMrMbMajbn0rzJsRjcdJNuMxIRkTaJcsCVJUlzCds/A6qcc3VmybIHBzlXDVQDjBw5MvEc2RGLwdix/jajsjI/+Ep9vyIikoIog+96YFBo+0BgQ0KekcBDQeDtD3zBzGqdc/MiLFd6JLvPV8FXRERSEGXwfQ4YZmZDgLeBCcCF4QzOuSHx12Z2PzA/LwIvNNznG6/56j5fERFJUWTB1zlXa2ZX40cxFwP3OueWm9mUYH+L/bw5T/f5iohIO5lzudGFmqqRI0e6mpqabBdDRCSvmNnzzrmR2S6HeJpesr000llERNpJ00u2h0Y6i4hIB6jm2x5a0UhERDpAwbc9tKKRiIh0gJqd20MjnUVEpAMUfNurslJBV0RE2kXNziIiIhmm4CsiIpJhCr7tpft8RUSkndTn2x66z1dERDpANd/20H2+IiLSAQq+7aH7fEVEpAPU7Nweus9XREQ6QDVfERGRDFPNtz004EpERDpANd/2mDMHPvpIA65ERKRdFHzbKhaDe+8F5/x2SYkGXImISJso+LbVokVQW+tfm8Fll6nJWURE2kTBt6369YP6ev/aOTjuuOyWR0RE8o6Cb1v98Y8Nr83g3XezVxYREclLCr5tUV0N8+Y1bDvna8IiIiJtoODbFg8/3DRNNV8REWmjSIOvmZ1hZqvNbI2ZTUuyf5yZvWRmS82sxsw+E2V5Omz8+MbbpaUa6SwiIm0WWfA1s2LgDuBMYDjwFTMbnpBtIXCMc+5Y4KvAPVGVJ20OPhj69oVTToHFizXSWURE2izKGa5GAWucc68DmNlDwDhgRTyDc25HKH93wEVYno6probJkxu2tY6viIi0U5TNzgOBt0Lb64O0RszsPDNbBfwBX/ttwswmBc3SNZs3b46ksC2qrobp0xun7d2rma1ERKRdogy+liStSc3WOfeoc+5w4FzgR8lO5Jyrds6NdM6NrKioSG8pWxOv8b73XuP04mL194qISLtEGXzXA4NC2wcCG5rL7Jx7GjjUzPpHWKa2SzbCGWDECPX3iohIu0QZfJ8DhpnZEDMrAyYAj4UzmNlQM7Pg9fFAGZBb9+4kjnCOu/zyzJZDREQ6jciCr3OuFrgaeBJYCfzWObfczKaY2ZQg23jgZTNbih8Z/W/OuWgGXcVicN55cOKJvik5FVVV8B//0TT9iCNg0qT0lk9ERApGpOv5OuceBx5PSLs79HomMDPKMgA+8J5ySsOCCM8+659bCqBVVTBrVvJ9Z5+d3vKJiEhBiTT45ozwSkRxkyfD178ORUV+oQTn/FzN4F/X1TV/vj59oiqpiIgUgMIIvmPG+NHJiQE1MSCnoksXjXIWEZEOKYy5nSsr4corO3aOoUNhyhT4y180yllERDqkMIIvwMUX+9pve0ydCq++CnfdpcArIiIdVjjBt7ISlizxA6+6dfOBuKQEysr8c3w7/LpPHx94Z0Y/JkxERApHYfT5xlVW+sUQREREsqhwar4iIiI5QsFXREQkwxR8RUREMkzBV0REJMMUfEVERDJMwVdERCTDLKpFhKJiZpuBN9t5eH9gSxqLkw90zYVB11wYOnLNBzvnKtJZGGm/vAu+HWFmNc65kdkuRybpmguDrrkwFOI1d1ZqdhYREckwBV8REZEMK7TgW53tAmSBrrkw6JoLQyFec6dUUH2+IiIiuaDQar4iIiJZp+ArIiKSYQUTfM3sDDNbbWZrzGxatsuTDmY2yMz+YmYrzWy5mX0rSN/HzP5kZq8Gz31Dx0wPPoPVZnZ69krfMWZWbGYvmtn8YLtTX7OZ9TGz35nZquDnXVkA13xt8L1+2cx+Y2ZdO9s1m9m9ZvaOmb0cSmvzNZrZCDP7Z7Dvv8zMMn0t0kbOuU7/AIqB14BDgDJgGTA82+VKw3XtDxwfvO4JvAIMB2YB04L0acDM4PXw4Nq7AEOCz6Q429fRzmv/D+BBYH6w3amvGfgVcEXwugzo05mvGRgIvAF0C7Z/C1za2a4ZOAU4Hng5lNbmawSeBSoBA/4InJnta9Oj5Ueh1HxHAWucc6875/YADwHjslymDnPObXTOvRC83g6sxP/RGof/Y03wfG7wehzwkHNut3PuDWAN/rPJK2Z2IPBF4J5Qcqe9ZjPrhf8j/UsA59we59xWOvE1B0qAbmZWApQDG+hk1+ycexp4LyG5TddoZvsDvZxzMecj8ZzQMZKjCiX4DgTeCm2vD9I6DTMbDBwHPAPs55zbCD5AA/sG2TrL5/AzYCpQH0rrzNd8CLAZuC9oar/HzLrTia/ZOfc2cAuwDtgIfOCce4pOfM0hbb3GgcHrxHTJYYUSfJP1f3Sae6zMrAfwMHCNc25bS1mTpOXV52BmZwHvOOeeT/WQJGl5dc34GuDxwF3OueOAnfjmyObk/TUH/Zzj8M2rBwDdzWxiS4ckScura05Bc9dYCNfe6RRK8F0PDAptH4hvwsp7ZlaKD7xznXOPBMn/CpqiCJ7fCdI7w+fwaeAcM1uL7z441cx+Tee+5vXAeufcM8H27/DBuDNf82nAG865zc65vcAjwEl07muOa+s1rg9eJ6ZLDiuU4PscMMzMhphZGTABeCzLZeqwYETjL4GVzrmfhnY9BlwSvL4E+H0ofYKZdTGzIcAw/ECNvOGcm+6cO9A5Nxj/c/yzc24infuaNwFvmdkngqSxwAo68TXjm5s/ZWblwfd8LH5MQ2e+5rg2XWPQNL3dzD4VfFYXh46RXJXtEV+ZegBfwI8Gfg24PtvlSdM1fQbfvPQSsDR4fAHoBywEXg2e9wkdc33wGawmz0dEAmNoGO3cqa8ZOBaoCX7W84C+BXDNPwBWAS8DD+BH+XaqawZ+g+/T3ouvwV7enmsERgaf02vA7QSzF+qRuw9NLykiIpJhhdLsLCIikjMUfEVERDJMwVdERCTDFHxFREQyTMFXREQkwxR8RQJmVmdmS0OPaUH6omAVmWVm9rf4/bZmVmZmPzOz14IVaH4fzDsdP98AM3so2L/CzB43s8PMbHB4FZsg7w1mdl1mr1hEsqUk2wUQySG7nHPHNrPvIudcjZlNAm4GzgFuxK8mdZhzrs7MLgMeMbMTg2MeBX7lnJsAYGbHAvvReH5eESlACr4ibfM0cI2ZlQOXAUOcc3UAzrn7zOyrwKn4yU/2Oufujh/onFsKHy+CISIFTMFXpEE3M1sa2r7JOfc/CXnOBv4JDAXWuaYLWdQARwavW1r84dCE9xqAX8VHRAqAgq9Ig5aaneea2S5gLfANYB+SrxxjNL/STNhr4fcysxvaWFYRyWMKviKpucg5VxPfMLP3gIPNrKdzbnso3/HA/wWvL8hkAUUkf2i0s0g7OOd2Ar8CfmpmxQBmdjFQDvw5eHQxsyvjx5jZCWY2OhvlFZHcouAr0qBbwq1GM1rJPx34CHjFzF4FvgSc5wLAecDngluNlgM3oHVWRQS0qpGIiEimqeYrIiKSYQq+IiIiGabgKyIikmEKviIiIhmm4CsiIpJhCr4iIiIZpuArIiKSYf8ftXJOpPo34msAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(6,4))\n",
    "\n",
    "#plt.plot(accuracies)\n",
    "\n",
    "plt.plot( train_accuracy, '.b', label = 'training')\n",
    "plt.plot( test_accuracy, '.r', label = 'testing')\n",
    "\n",
    "axes = plt.gca()\n",
    "plt.legend(bbox_to_anchor=(1.04,1), loc=\"upper left\")\n",
    "axes.set_xlabel('EPOCH')\n",
    "axes.set_ylabel('accuracy')\n",
    "plt.title('Normalized data accuracy', fontsize = 16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the accuracy of the final iterations:\n",
    "\n",
    "Un-normalized, \n",
    "$$(\\% accuracy)_{train} = 69.2\\%$$\n",
    "$$(\\% accuracy)_{test} = 54.2\\%$$\n",
    "Normalized,\n",
    "$$(\\% accuracy)_{train} = 90.8\\%$$\n",
    "$$(\\% accuracy)_{test} = 66.6\\%$$\n",
    "We can observe that the normalized data has a much better percent accuracy than the un-normalized case. \n",
    "Additionally, from the Accuracy plots the updating of accuracies is a much smoother process with the normalized data. \n",
    "\n",
    "Recall with the Perceptron model on the same data, we had accuracies of:\n",
    "\n",
    "Un-normalized, \n",
    "$$(\\% accuracy)_{train} = 64.8\\%$$\n",
    "$$(\\% accuracy)_{test} = 68.0\\%$$\n",
    "Normalized,\n",
    "$$(\\% accuracy)_{train} = 77.0\\%$$\n",
    "$$(\\% accuracy)_{test} = 81.7\\%$$\n",
    "So we observe that in both cases, the training accuracies have increased by using a Neural Network. Although, now the testing accuracies have decreased, and are lower than the training accuracy. Because they are close, it's a good sign that we're not overfitting the data. In other words, we can apply our network to new data the network hasn't seen before, and it will perform fairly well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}