{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sheet 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us first set up some basic functions, including a few from sheet 1, that we will use througout the worksheet. We begin by importing some necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "import scipy.spatial.distance as dist\n",
    "import matplotlib.pyplot as plot\n",
    "import colorsys\n",
    "import os\n",
    "import gzip\n",
    "from urllib.request import urlretrieve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Secondly, we need basic functions to create a linear classifier (from sheet 1) and to create classifiers from models mapping to real numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepend_ones(matrix):\n",
    "    \"\"\"Adds a column of ones on the left of the given matrix\n",
    "    :param matrix: matrix to append column to.\n",
    "    \n",
    "    :return: ndarray. given matrix with additional column of ones on the left.\n",
    "    \"\"\"\n",
    "    return np.append(np.ones((len(matrix), 1)), matrix, axis = 1)\n",
    "    \n",
    "\n",
    "def lls(dataset, labels):\n",
    "    \"\"\"Computes a multivariate linear function minimizing squared errors\n",
    "    \n",
    "    :param dataset: n times d numpy matrix containing n samples from d-dimensional domain of function to be learned \n",
    "    :param labels: function values corresponding to sample data in dataset\n",
    "    \n",
    "    :return: ndarray. linear function minimizing squared errors shaped as vector of dimension d + 1\n",
    "    \"\"\"    \n",
    "    X = prepend_ones(dataset)\n",
    "    return np.linalg.solve(np.dot(X.T, X), np.dot(X.T, labels))\n",
    "\n",
    "def linear_function(alpha):\n",
    "    \"\"\"Transforms vector into a multivariate linear function\n",
    "    :param alpha: d + 1 dimensional vector\n",
    "    \n",
    "    :returns: function. linear function from d dimensional domain into the real numbers\n",
    "    \"\"\"\n",
    "    def func(x):\n",
    "        if (len(np.shape(x)) == 1):\n",
    "            return alpha[0] + np.dot(alpha[1:], x)\n",
    "        else:\n",
    "            X = prepend_ones(x)\n",
    "            return np.dot(X, alpha)\n",
    "    return func\n",
    "\n",
    "def create_classifier(model, classes, borders):\n",
    "    \"\"\"Turns a model mapping to real numbers into a classifier based on given classes and borders\n",
    "    \n",
    "    :param model: a function mapping the input space to the real numbers\n",
    "    :param classes: classes range the output model should map to\n",
    "    :param borders: array of real numbers of to determine borders between the classes in orders\n",
    "    \n",
    "    :returns: function. mapping input space to range of classes array based on borders\n",
    "    \"\"\"\n",
    "    if len(classes) != len(borders) + 1:\n",
    "        raise Exception(\"Number of classes and borders does not match\")\n",
    "    def f(x):\n",
    "        if len(np.shape(x)) == 1:\n",
    "            return classes[sum(model(x) >= borders)]\n",
    "        else:\n",
    "            return np.array([f(y) for y in x])\n",
    "    return f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third block of function definitions sets up standard methods to evaluate the accuracy of a model (also from sheet 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conf_matrix(labels, predictions):\n",
    "    \"\"\"Calculates the confusion matrix given the actual and predicted labels\n",
    "    :param labels: actual labels\n",
    "    :param predictions: predicted labels\n",
    "    \n",
    "    :return: ndarray. confusion matrix\n",
    "    \"\"\"\n",
    "    label_list = np.unique(labels)\n",
    "    return np.array([[np.count_nonzero(np.logical_and(labels == label_list[i], predictions == label_list[j])) \n",
    "                      for j in range(len(label_list))]\n",
    "                      for i in range(len(label_list))])\n",
    "\n",
    "def accuracy(labels, predictions):\n",
    "    \"\"\"Calculates the accuracy of a model given the actual and predicted labels\n",
    "    \n",
    "    :param labels: actual labels\n",
    "    :param predictions: predicted labels\n",
    "    \n",
    "    :return: number. accuracy\n",
    "    \"\"\"\n",
    "    return np.trace(conf_matrix(labels, predictions)) / len(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define a revised version of the plot function that we wrote for sheet 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_color_palette(count):\n",
    "    \"\"\"Generates evenly distributed rgb colours\n",
    "    \n",
    "    :param count: number of colours to generate\n",
    "    \n",
    "    :returns: list. containing evenly distributed colours\n",
    "    \"\"\"\n",
    "    def to_int_rgb(tup):\n",
    "        return tuple([int(255 * x) for x in tup])\n",
    "\n",
    "    return ['#%02x%02x%02x' % to_int_rgb(colorsys.hsv_to_rgb(*(x*1.0/count, 0.5, 0.5))) for x in range(count)]\n",
    "\n",
    "def plot_contour_2d_classification(data, labels, model, grid_delta = 0.1, title = \"\", additional = None):\n",
    "    \"\"\"Plot contour of a 2d classifier\n",
    "    :param data:\n",
    "    :param labels:\n",
    "    :param model: function expecting an array of d dimensional arrays returning a value\n",
    "    :param grid_delta: resolution of the grid\n",
    "    :param title: title of the plot\n",
    "    :additional: function returning a dictionary containing of tuples of legend keywords and plot objects to be plotted\n",
    "    \"\"\"\n",
    "    overlap = 0.5\n",
    "    x_range = (min(data[:, 0]) - overlap, max(data[:, 0]) + overlap)\n",
    "    y_range = (min(data[:, 1]) - overlap, max(data[:, 1]) + overlap)\n",
    "\n",
    "    x, y = np.meshgrid(np.arange(*x_range, grid_delta), np.arange(*y_range, grid_delta))\n",
    "    \n",
    "    z = np.zeros(np.shape(x))\n",
    "    for i in range(np.shape(x)[1]):\n",
    "        array = np.array([x[:, i], y[:, i]]).T\n",
    "        z[:, i] = model(array)\n",
    "        \n",
    "    label_list = np.unique(labels)\n",
    "    rgb_colors = generate_color_palette(len(label_list))\n",
    "    rgba_colors = [color + \"44\" for color in rgb_colors]\n",
    "    \n",
    "    plot.rc('text', usetex=True)\n",
    "    plot.figure()\n",
    "    \n",
    "    cp = plot.contourf(x, y, z, 1, colors = rgba_colors)\n",
    "    \n",
    "    np.shape(data)\n",
    "    \n",
    "    class_dict = dict()\n",
    "    for i in range(len(label_list)):\n",
    "        class_dict['Class ' + str(int(label_list[i]))] = plot.scatter(*zip(*(data[np.equal(labels, label_list[i]),:])),\n",
    "                                                                      color = rgb_colors[i])\n",
    "\n",
    "    plot.title(title, fontsize = 10, color = 'black')\n",
    "    plot.xlabel(r'$x_1$')\n",
    "    plot.ylabel(r'$x_2$')\n",
    "            \n",
    "    plots = list(class_dict.values())\n",
    "    keys = list(class_dict.keys())\n",
    "    \n",
    "    additional_keys, additional_plots = [], []\n",
    "    if additional != None:\n",
    "        additional_dict = additional()\n",
    "        additional_keys, additional_plots = list(additional_dict.keys()), list(additional_dict.values())\n",
    "        \n",
    "    if len(keys) != 0:\n",
    "        for key in additional_keys:\n",
    "            if key in keys:\n",
    "                raise Exception(\"Multiple plots for the same key\")\n",
    "        plots += additional_plots\n",
    "        keys += additional_keys\n",
    "\n",
    "    plot.legend(plots, keys)\n",
    "    \n",
    "    plot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.1\n",
    "The first task is to implement the **OneStep** algorithm, which is a single step of the sequential minimal optimization algorithm to train a support vector classifier. Because it is not explicitly prohibited, we implement the algorithm to work with a kernel function right from the start that we set to the standard dot product by default. This way, we do not need to make any adaptions later on and keep our code modular and easier to read and maintain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def standard_kernel(t, X):\n",
    "    \"\"\"Standard dot product\n",
    "    :param t: matrix or vector from input domain\n",
    "    :param X: matrix from input domain\n",
    "    \"\"\"\n",
    "    return np.dot(t, X.T)\n",
    "\n",
    "def create_svm_evaluater(X, y, beta, b, C, kernel):\n",
    "    \"\"\"Creates an svm evaluation function\n",
    "    :param X: training set\n",
    "    :param y: labels\n",
    "    :param beta: beta parameter of svm optimization problem\n",
    "    :param b: b parameter of svm optimization problem\n",
    "    :param C: regularization parameter\n",
    "    :param kernel: Kernel function for sets X, Y. Returns Matrix M with M_ij = K(X_i, Y_j) \n",
    "    \n",
    "    :returns: svm function corresponding to given parameters\n",
    "    \"\"\"\n",
    "    def f(x):\n",
    "        if len(np.shape(x)) == 1:\n",
    "            return np.sum(beta * y * kernel(x, X)) + b\n",
    "        else:\n",
    "            return np.sum(beta.reshape((len(beta), 1)) * y.reshape((len(y), 1)) * kernel(X, x), axis = 0) + b\n",
    "    \n",
    "    f.support_vectors = X[beta > 0]\n",
    "    f.margin_def_vectors = X[np.logical_and(np.array(beta > 0), np.array(beta < C))]\n",
    "    f.X = X\n",
    "    f.y = y\n",
    "    f.beta = beta\n",
    "    f.b = b\n",
    "    f.C = C\n",
    "    f.kernel = kernel\n",
    "    \n",
    "    return f\n",
    "\n",
    "def OneStep(X, y, C, beta, b, i, j, kernel):\n",
    "    \"\"\"Performs one step of the Sequential minimal optimization algorithm\n",
    "    :param X: training set\n",
    "    :param y: labels\n",
    "    :param C: regularization parameter\n",
    "    :param beta: beta parameter of svm optimization problem\n",
    "    :param b: b parameter of svm optimization problem\n",
    "    :param i: first index to optimize on\n",
    "    :param j: second index to optimize on\n",
    "    :param kernel: Kernel function for sets X, Y. Returns Matrix M with M_ij = K(X_i, Y_j). Returns an array if\n",
    "                   X is a single example\n",
    "    \n",
    "    :returns: tuple. one step optimized (beta, b)\n",
    "    \"\"\"\n",
    "    f = create_svm_evaluater(X, y, beta, b, C, kernel)\n",
    "    \n",
    "    fVals = f(X[[i, j]])\n",
    "    fXi = fVals[0]\n",
    "    fXj = fVals[1]\n",
    "    \n",
    "    delta = y[i] * ((fXj - y[j]) - (fXi - y[i]))\n",
    "    s = y[i] * y[j]\n",
    "    chi = kernel(X[i], X[i]) + kernel(X[j], X[j]) - 2 * kernel(X[i], X[j])\n",
    "    gamma = s * beta[i] + beta[j]\n",
    "    \n",
    "    L, H = (max(0, gamma - C), min(gamma, C)) if s == 1 else (max(0, -gamma), min(C, C - gamma))\n",
    "    \n",
    "    if chi > 0:\n",
    "        beta[i] = min(max(beta[i] + delta / chi, L), H)\n",
    "    elif delta > 0:\n",
    "        beta[i] = L\n",
    "    else:\n",
    "        beta[i] = H\n",
    "    \n",
    "    beta[j] = gamma - s * beta[i]\n",
    "    \n",
    "    b = b - 0.5 * (fXi - y[i] + fXj - y[j])\n",
    "    return beta, b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.2\n",
    "After having implemented the **OneStep** algorithm, we create a little data set to test our algorithm on before finishing the whole algorithm in the following task. The dataset contains 40 examples, 20 drawn from the exponential distribution with parameter 4 labelled -1 and 20 drawn from the exponential distribution with parameter 0.5 labelled 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataset(N, params):\n",
    "    \"\"\"Creates a dataset consisting of 2N two dimensional examples drawn from the exponential distribution\n",
    "    :param N: Number of examples to draw from each exponential distribution\n",
    "    :param params: Exactly 2 parameters for the exponential distribution to draw from.\n",
    "    \n",
    "    :returns: ndarray. (2*N, 2) matrix. First N labelled -1, second N labelled 1.\n",
    "    \"\"\"\n",
    "    if N <= 0 or len(params) != 2:\n",
    "        raise Exception(\"N < 0 or parameters have wrong size\")\n",
    "    \n",
    "    def create_single(N, param, label):\n",
    "        if param <= 0:\n",
    "            raise Exception(\"lambda <= 0\")\n",
    "        return np.concatenate((np.random.exponential(1 / param, (N, 2)), label * np.ones((N,1))), axis = 1)\n",
    "    \n",
    "    return np.concatenate((create_single(N, params[0], -1), create_single(N, params[1], 1)), axis = 0)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "N = 20\n",
    "exp_params = [4, 0.5]\n",
    "\n",
    "small_set = create_dataset(N, exp_params)\n",
    "X = small_set[:, :(-1)]\n",
    "y = small_set[:, -1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.3\n",
    "Now we implement the sequential minimal optimization algorithm **SMO** using the **OneStep** function we wrote earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def SMO(X, y, C, max_iter, kernel = standard_kernel):\n",
    "    \"\"\"Trains a support vector machine using sequential minimal optimization\n",
    "    :param X: Traing set\n",
    "    :param y: labels for the training set\n",
    "    :param C: regularization parameter\n",
    "    :param max_iter: number of iterations to run\n",
    "    :param kernel: Kernel function for sets X, Y. Returns Matrix M with M_ij = K(X_i, Y_j). Returns an array if\n",
    "                   X is a single example\n",
    "    \n",
    "    :returns: support vector machine for 2-class classification\n",
    "    \"\"\"\n",
    "    n = len(X)\n",
    "    beta = np.zeros(len(X))\n",
    "    b = 0\n",
    "    \n",
    "    for k in range(max_iter):\n",
    "        i, j = np.random.choice(n, 2, replace = False)\n",
    "        beta, b = OneStep(X, y, C, beta, b, i, j, kernel)\n",
    "    \n",
    "    f = create_svm_evaluater(X, y, beta, b, C, kernel)\n",
    "    b = b - np.ma.median(f(X))\n",
    "    \n",
    "    return create_svm_evaluater(X, y, beta, b, C, kernel)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After having implemented the **SMO** we now want to compare the performance of our models for different values for the regularization parameter *C* with a linear classifier from sheet 1. Therefore, using our algorithms, we train four models on the training set we generated in an earlier task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_iter = 10000\n",
    "small_C = 0.1\n",
    "medium_C = 1\n",
    "big_C = 100\n",
    "\n",
    "svm_smallC = SMO(X, y, small_C, max_iter)\n",
    "svm_mediumC = SMO(X, y, medium_C, max_iter)\n",
    "svm_bigC = SMO(X, y, big_C, max_iter)\n",
    "lls_model = linear_function(lls(X, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before looking at the results, we write a few functions to examine our support vector classifiers. We are interested in the support vectors and margin defining vectors used by the classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_support_vectors(svm, color, symbol, col_idx = [0, 1]):\n",
    "    \"\"\"Adds support vectors to a plot.\n",
    "    :param svm: support vector machine to take support vectors from\n",
    "    :param color: color of the mark the support vectors in the plot\n",
    "    :param symbol: symbol to mark the support vectors in the plot\n",
    "    :param col_idx: features indices to plot\n",
    "    \n",
    "    :returns: plot object.\n",
    "    \"\"\"\n",
    "    return plot.scatter(*zip(*svm.support_vectors[:, col_idx]), color = color, marker = symbol, s = 10)\n",
    "\n",
    "def add_margin_def_vectors(svm, color, symbol, col_idx = [0, 1]):\n",
    "    \"\"\"Adds margin defining vectors to a plot.\n",
    "    :param svm: support vector machine to take margin defining vectors from\n",
    "    :param color: color of the mark the margin defining vectors in the plot\n",
    "    :param symbol: symbol to mark the margin defining vectors in the plot\n",
    "    :param col_idx: features indices to plot\n",
    "    \n",
    "    :returns: plot object.\n",
    "    \"\"\"\n",
    "    return plot.scatter(*zip(*svm.margin_def_vectors[:, col_idx]), color = color, marker = symbol, s = 10)\n",
    "    \n",
    "def add_important_vectors(svm, col_idx = [0, 1]):\n",
    "    \"\"\"Adds support vectors and margin defining vectors to a plot\n",
    "    :param svm: support vector machine to take the vectors from\n",
    "    :param: features indices to plot\n",
    "    \n",
    "    :retunrs: dictionary. Contains the plots for both support vector machines and the margin defining vectors.\n",
    "    \"\"\"\n",
    "    sup = add_support_vectors(svm, \"white\", \"D\", col_idx)\n",
    "    mar = add_margin_def_vectors(svm, \"black\", \"x\", col_idx)\n",
    "    return {\"Support Vectors\": sup, \"Margin Vectors\": mar}\n",
    "\n",
    "def print_important_vectors(svm, title):\n",
    "    \"\"\"Prints the number of support vectors and margin defining vectors\n",
    "    :param svm: support vector machine to take the vectors from\n",
    "    :param title: name of the svm\n",
    "    \"\"\"\n",
    "    print(\"The \" + title + \" has \" + str(len(svm.support_vectors)) + \" support vectors and \"\n",
    "          + str(len(svm.margin_def_vectors)) + \" margin defining vectors.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now convert our models into classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = [-1, 1]\n",
    "borders = [0]\n",
    "delta = 0.02\n",
    "\n",
    "lls_classifier = create_classifier(lls_model, classes, borders)\n",
    "svm_smallC_classifier = create_classifier(svm_smallC, classes, borders)\n",
    "svm_mediumC_classifier = create_classifier(svm_mediumC, classes, borders)\n",
    "svm_bigC_classifier = create_classifier(svm_bigC, classes, borders)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having all tools available now, we can take a look at our classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAERCAYAAACepNcKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGYFJREFUeJzt3W1oXOeZxvHrtizZriA7luttNg1sPck2xMWgSDK43S1pvcoHYQJpVmlpWdpsDTIUZ2Gb1GFx0y9tKXHqDd34S9QX0mwJeKOmLbtFFKtuQyER1HIEfQlpyTiloQt2LI9dVMcv0r0fdGYyGo9GZ0bn6LzM/wfBmqOZM0/E+NLj+zznfszdBQDIpw1JDwAAEB9CHgByjJAHgBwj5AEgxwh5AMgxQh6pZ2av1z0eNrOn1/H9i2b2eJuvrY7VzA6Z2dPBsUPRjhJobGPSAwBa5e5TkqaSHkcYdWPd7e4PBF9nYvzIPmbywPopJD0AdB5CHplTKZ8Ef54ISiAzZlYIvv98cHzUzAo1j8fqzvF08L1ig/d4OnjNiZpjN5wrKL08Hzx/oMHjylgPSRqqOf54g7E2HRPQDso1yDx3PxCE5pCZDUg64e7jZva8u09IekCSzGxG0njNSz8uaYe7l2vPFwT46+5+IHhcDN6n3OBcD0j6mrufDo4/Xfe48tojZnZPMNaB4HuHascq6fRKYwLaRcgj60rBn+e1VA65TVKfmQ1KKgSz+zFJ2yTVz47/e4UwHZTU8MJuEMy153pc0uPB+xxo8LiZZWOVdFOTMQFtIeSRNzOS5oIZfCWUS8FMerTuuSuF6YykYS3NrKsancvdS5IeMLNhSQfc/dHax1rhl8UKYy02GRPQFkIeWVCsqY2XJT260hMrpQ8zOxA892uSvmlm94R9s+AcTwfvWft+U/XnCoK/8vjR+sch3qd2rE2fD7TD6EIJAPnF6hoAyDFCHgByjJAHgBwj5AEgxwh5AMixxJdQFvr6/Jb3vjfpYQBAprz661+/5e7bV3te4iF/y3vfq2d/+MOkhwEAmbL79tv/EOZ5lGsAIMcIeQDIMUIeAHIs8Zo8ACyzuKiuq1dli4tSp7ddMZNv2KCFnh5pQ3tzckIeQKp0Xb2qv+7r018VCjKzpIeTKHfXxXJZZ+fmtLB5c1vnoFwDIFVscZGAD5jZ0s9icbHtcxDyANLFPRUB/9jhw7p33z7986c+pSePHtWZM2f02OHDsbzXQwcPrvg9M1tT2YqQB4A6Dx08qB3Fov7nxz/W9557Tv+yf38s71Mul/XQwYP6wfe/H8v5JWryADLutydP6hfPPKNL587ppu3b9eEHH9TOvXvbPl+5XNbPT57UU8eOVY8VCgVduHAhiuEuUygU9NSxY3rl9OnVn9wmQj4jZkolTc7Oqjw/r0Jvr0b6+zVYrN+yFOgsvz15Uj/5xjd0/coVSdKls2f1k298Q5LaDvpXTp/WR5q8tlwu6+DnPqeLFy/qY/ffr8/u36+fnTypb3/rW9ra16f9+/frwoULyx7333VXW2OJAiGfATOlkiamp3VtYUGSVJ6f18T0tCQR9Ohov3jmmWrAV1y/ckW/eOaZNc3mmykUCvrec89Jkv7hQx/SZ/fv1wsvvKBHvvCFapg/dPDgssdJIuQzYHJ2thrwFdcWFjQ5O0vIo6NdOneupeNh3DUwoH9tciFUkp48elRzc3N648wZSdLnH35Yjx0+rIsXL+o/jx274fGOHTuqr33s8GGVL15UcccO/dvDD7c9zrC48JoB5fn5lo4DneKm7Y2bMK50PIxCoaCP7N2r73z729Vj5XK5+vWTR4/qfTt26Mtf/aq2bt0qSdqxY4e+99xz+vzDD+s73/rWDY9rffmrX9VTx46tS8BLhHwmFHp7WzoOdIoPP/igNm7atOzYxk2b9OEHH1zTeZ86dkxnSiXdu2+f7t23T0efeKL6vY/u3aujX//6smWPTx49qnv37dN/HD2qfxodveFxMw8dPKg3zpzRQwcP6kzwL4MomSd82/DOXbucVsPN1dfkJam7q0uje/ZQrkHubJyf19+9//2hnx/16po0+v3vfqfrdZO63bffPuPuQ6u9NpaavJkNSCpKkrtPxPEenaQS5KyuAW60c+/e3IV6lOK68HrA3Q+Y2SEzG3D3+BaBdojBYpFQB9CyyEPezEYlvS5J7n4k6vMDAMKL48LrbknbzGzAzA7FcH4AQEhxra45XynRBDN7AEAC4gj51yWVgq9LWprZL2NmY2Z2ysxOXZibi2EIALA2cXehbNZ5MkpxhPyUgpU1wZ+/rH+Cu4+7+5C7D23t64thCADQvji7UK5H58lakV94dfeSmZUrZRqWUAKI089+8xt998UX9dalS3r3TTfpM3ffrY9+4ANtny/uLpTr0XmyVixLKN19PI7zAkCtn/3mN3pqclJXrl+XJJ27dElPTU5KUttBTxdKAEiJ7774YjXgK65cv67vvvjimmbzzWStCyW9awBk1luXLrV0PIy7Bgb085Mnmz7nyaNH9djhw8u6UH79iSd07759OnPmzA2Pk0TIA8isd990U0vHw4i7C+V6I+QBZNZn7r5bmzYurzpv2rhRn7n77jWdN+4ulHF3nqxFF0oAqdJqF8qoV9ekUeq6UALAevnoBz6Qu1CPEuUaAMgxQh4AcoyQB5AuZkr6WmGauLtk1vbrCXkAqeIbNuhiuUzQayngL5bL8g3tRzUXXgGkykJPj87OzencW29JnR70ZvING7TQ09P2KQh5AOmyYYMWNm9OehS5QbkGAHKMkAeAHCPkASDHCHkAyDEuvAKomimVNDk7q/L8vAq9vRrp79dgsbj6C5FahDwASUsBPzE9rWsLC5Kk8vy8JqanJYmgzzDKNQAkSZOzs9WAr7i2sKDJ2dmERoQoEPIAJC3N3Fs5jmwg5AFIkgp1/cpXO45sIOQBSJJG+vvV3dW17Fh3V5dG+vsTGhGiEEvIm9njwZ9jcZwfQPQGi0WN7tlTnbkXens1umcPF10zLq7VNWNmNirpQEznBxCDwWKRUM+ZuEL+AXefiuncAICQ4gr5AVtqcj/g7kdieg8AwCpiqcm7+5FgJr/NzIbrv29mY2Z2ysxOXZibi2MIAADFMJMPavFy9wlJ5yXdUOBz93FJ45K0c9euDt8VAPW4tR6IThzlmlLwnyTdJunpGN4DOcWt9UC0Ii/XuPtpSR8PZvSvB4+BULi1HohWLBdeg3IM0DJurQeixR2vSBVurQeiRcgjVbi1HogW/eSRKpWLq6yuAaJByCN1uLUeiA7lGgDIMUIeAHKMkAeAHCPkASDHCHkAyDFCHgByjJAHgBwj5AEgxwh5AMgxQh4AcoyQB4AcI+QBIMcIeQDIMUIeAHKMVsMtmimV6HUOIDMI+RbMlEqamJ6ubjRdnp/XxPS0JBH0AFIp1nKNmR2K8/zrbXJ2thrwFdcWFjQ5O5vQiACgudhC3syGJe2O6/xJKM/Pt3QcAJLGhdcWbOnpaXi80Nu7ziMBgHBiCXkzG3D3qTjOnZSZUklXr1+/4fgGM4309ycwIgBYXVwz+b6YzpuYydlZLSwu3nB8c08PF10BpFbkIR9mFm9mY2Z2ysxOXZibi3oIsVip7v6XK1fWeSQAEF4cM/mimY2a2Wjw9UD9E9x93N2H3H1oa182Jv0r1d2pxwNIs8hD3t0n3H1CSyWbQtTnT8pIf7+6u7qWHevu6qIeDyDVYrsZyt3HJY3Hdf71Vqm7c7crgCzhjtcWDBaLhDqATGGdPADkGCEPADlGyANAjhHyAJBjhDwA5BghDwA5RsgDQI4R8gCQY4Q8AOQYIQ8AOUbIA0COEfIAkGOEPADkGCEPADlGyANAjhHyAJBjhDwA5FiokDezHWb2iJm9L3h8f5yDAgBEI+xMflTSTyU9YGb9ku6Jb0gAgKiE3eN1yt1fkfSKmf2jJDY6BYAMCBvyZTO7391fcPefmlmsgwIARGPVkDezm9z9jKQzlWPu/tNVXjMcfHmPuz+6tiECANoVpib/70EdXmZ2V+XrlZjZgJbCfUrSQPAYAJCAMOWaU5KKZlZy91fMbG+zJ7v7aUmng4fF4DEAIAFhQr4oqSzpiJntkHRC0snVXmRmhyQdWNvwAOTJTKmkydlZlefnVejt1Uh/vwaLrOOIU5iQL7n79yV9Uwq/Rt7dj5jZ82Z2yt3Ltd8zszFJY5J08y23tDjkfOLDj7ybKZU0MT2tawsLkqTy/Lwmpqclic96jFatybv792tugrpL0m3Nnm9mtXX4koIwrzvnuLsPufvQ1r6+lgedN5UPf3l+XtI7H/6ZUinhkQHRmZydrQZ8xbWFBU3OziY0os4Q6mYod38j+PMVd39ilacPS6okd0FLQY8m+PCjE1QmMWGPIxpx9K4Z19KF2lFJcveJGN4jV/jwoxMUentbOo5oRB7y7l4OyjET7s6F1xD48KMTjPT3q7ura9mx7q4ujfQ3XZWNNaILZQrw4UcnGCwWNbpnT3XyUujt1eiePVx0jVnYtgaIUeVDHvXqGlbsIG0Gi0U+g+uMkE+JqD/8LFcDIBHykUvL7LnZih1CHugchHwLVgvwNM2eWbEDQOLCa2hhblhK03p3VuwAkAj50MIEeJpmz6zYASBRrgktTIAXensbPi+J2XNcK3YAZEviIX/t7bf1f7/7nSTpb97//oRHs7IwAT7S37+sJi8lO3tmuRqAxMs1m7u7tXP7dkmqhn0ahSl/cLMHgLRJfCZfsXP7dv323LllQZ+mmX3Y8gezZwBpkpqQl1Sd0UtaFvhpCXsCHEDWpCrka1UCP82zewBIu8Rr8qvZuX179T8p3XV7AEib1M7kG6nO7pnZA0AoqZ/JN1I/s2d2DwCNZWomX2/ZhVpm921JS0M1APHIdMjXanShlrBvLk0N1QDEIzchX9Fodk/YN0Y7YiD/chfytdJ+g1XS0tRQDUA8ch3yUvpvsEpS2IZq1O2B7Iol5M1sLPjyNnd/NI73aAd1++XCNFSjbo+oMFlIRuRLKM1sWNKUu49LKgaPU4UlmEvCNFRL00YoyK4wm+4gHnHM5IvBf+OSSsHXqUTrhNX78VC3RxS4yJ+cyEM+mMFXDEg6HvV7RI26/crStBEKsovJQnJiu+PVzAYknXD30w2+N2Zmp8zs1NyFC3ENoS2UcpZjG0FEgT2HkxNnW4Nhdz/S6BvuPu7uQ+4+1Ld1a4xDaB9hv4SNUBAFJgvJiW11TSXgzWzY3afieJ/1QOsE+uhj7dhzODmRh3ywmuZxM3tUUp+kB6J+j6SwBBNoH5OFZMRx4XVKUjprMBFhdg8gKzLZajhNqN0DSDNCPiKEPYA0yn3vmvXGDVYA0oSZfEzYmxZAGjCTXwe0PAaQFEJ+ndA6AUASKNckgIu0ANYLIZ8gwh5A3CjXpAA3VwGICyGfMrVLME++9JJ+8cc/6s9Xr9LrA0BbCPmUunjliqbeeENXa7bde/7llyWx7R6A8KjJp9QPXnutGvAV1xcX9b+//CW1ewChMZNPqfOXLzc8/uerVyUps0sw2cw5PH5WiAIhn1LbtmxpGPTbtmzJbOuEymbO12pKUBPT05IoQdXjZ4WoUK5JqY/dcYd66nbS6enq0sfuuKP6uFHrhDSXcppt5ozl+FkhKszkU+qDt94qaak2f/7yZW3bskUfu+OO6vF6WdjQhM2cw+NnhagQ8in2wVtvXTHUV5LmsC/09jYMKTZzvhE/K0SFck1ONSrjJF3KYTPn8PhZISrM5HOuUWO0pGb2bOYcHj8rRIWQ7yDVUk6CK3LYzDk8flaIQmzlGjMbiOvcWJssrcgBsDaxzOTNbFjS45IG4zg/okFjNCD/YpnJu/uUpLk4zo14MLsH8omaPJZpNLtnZg9kFyGPFbE3LZB9iYS8mY1JGpOkW26+OYkhICT2pgWyLZGbodx93N2H3H2ob+vWJIaANlC3B7InrtU1o5KGzGzU3SfieI9O9fKbb4buZxOXNLdOALBcLCEfBDvhHrGX33xTz/7qV9XNRM5fvqxnf/UrSVr3oJcah71E4ANpQu+aDGm0W9TVhQX94LXXEhrRkkYtjwGkA6trIhZnOWWl3aJWOp6ENLROAPAOQj5CcZdTmu0WlTbU7dGp0rZtI+WaCMVdTgmzW1Ta1JZy0tLyGIhLZdvGyl4AlW0bZ0qlxMZEyEegNDOjF77yFZ3/y18afj+qcsoHb71Vn961qzpz37Zliz69a1ciF13bwRJM5F0at23s+HJNaWZGs5OTmi+X1VsoqH9kRMXB8H3VSjMzmp6Y0MK1a9p05YqubN58w3OiLKe0s1tU2tA6AXmVxm0bO3omXwno+XJZkjRfLmt6YkKlmZnQ55idnNTCtWuSpGKppA11v8XTXk5JWtp2rwLWYqXtGZPctrGjQ742oCsWrl3T7ORk6HNUfkFI0nvOntUdr72mTW+/LblnrpySlEZLMAl7ZFEat23syHJNbYmmkflyWS985SuhSji9hcINQf+es2fVWyjo/i9+MZbx5xk3WCHL0rhtY8eFfG0NvZn6Eo6khkHfPzJyw/m6urvVPzIS4ag7T5r2pgVakbZtGzuuXNOoRLOaZiWc4uCg9oyOqrdQkLQ0s98zOtrSxVs0R90eaF/HzeRXKtGs5XXFwUFCPWa0PAba03Ez+cqMe71eh+hxkRYIr2Nm8qtdbG2GGns60ToBWF1HhHyzi629hYJuufNO/enVV6uraeoft3qD1EpjWMtNV1hZo5urJAIfkDok5Fe62Br3MseV/vWw2oodtI/ZPbBcbkM+THlmvlzWf3/pS7oS9Jzp2bJFu++7L5LgXW2pZmXFDiEfD1onAEtyGfJh18JLqga8JF29fFkvHT8uae0z7DBLNet/AaVha7882rl9OzdXoWPlcnVNO2vhK3xxsaW2BisJc4G3dsVOpRd9pWNlpRf9y2++ueaxgNYJ6Fy5nMmvFrA9W7boapP2v/Plsv7rkUfWdIG0vt1BvfoVO8160TObjxatE9BJcjmTb7am/e8/+Ul1b9oU6jztdKWs6B8ZUVd3d8PvbXrXu264KzYLW/vlDbN7dIJchnyzNe2trpVvtStlRaXdwaZ3veuG711vUEpaqed8Grf2yyPCHnkVS8ib2aiZDZvZoTjOv5pm5ZXKOvVWtNsKoTg4qI09PTccb/SLI4tb++URYY+8ibwmb2YDkuTuU2ZWNLMBdz8d9fs006y8UrnZ6fcvvxz6fGtpadCsnXGtSt19pdU13Ey1vqjbIy/iuPD6CUkngq9LkoYlrWvIn/rRj1b83i133qnSqVOhz7XWlgYrXYBt9Itjpa396peEcjPV+qHlMbIujnJNQdJczeNtMbxHU1dW2FBbkv706quhl1dG0Ta40QXYVn9xRLGDFdauUsqh5TGyJJEllGY2JmlMkm65+eZ1fe8w9fWu7u7IesJXzrGWUkvYkg/WB60TkCVxhHxZUl/wdUHS+fonuPu4pHFJ2rVzp0c9gJXWwfds2aLuTZsahqOZyd1jqXevtd98KyUfrB8aoyEL4gj545KGgq+LkqZieI+mdt93n146fly+uFg9Zhs2aPd990lSw+360rybE1sMpl/97J6gR1pEHvLuftrMhsxsWFJ5vVfWSOFKJFlaqRJFyQfroxr2zOyREuYeebWkJbt27vQfPvtsomMA4vLbc+eWPSbwEZXdt98+4+5Dqz0vl71rgLRgb1okjZAH1gk3WCEJuexdA6RZo8ZoQFyYyQMJYkMTxI2QBxJG3R5xolwDpAhdMBE1Qh5IIcIeUaFcA6QYrROwVszkgYxgRQ7awUweyBhaJ6AVibc1MLNzkv4Q41u8W9JbMZ4/Tlkeu5Tt8Wd57BLjT9J6jf1v3X37ak9KPOTjZmanwvR3SKMsj13K9vizPHaJ8ScpbWOnJg8AOUbIA0COdULIjyc9gDXI8tilbI8/y2OXGH+SUjX23Nfks8zMBpLYdAXZZ2aH3P1I0uNA8nI7kzezUTMbNrNDSY+lHcHOWt9MehztMLOx4L/Hkx5LO4LPzXCWxy9pd9LjaEflZ25mY0mPpR1mNhBkz2jSY6nIZcib2YAkufuUpHLlcZYEY59LehytCgJmKtisvRg8zozgs3JP8PMfyOJnJ+PGzOx1SaWkB9KmA+4+oaXPfio+O7kMeUmfkFQOvi5JylTQZFxR7/y8S8HjzHD30+7+aPCwmLVyWVDim0p6HGvwgLvflsX/h2D2/rokufuRtHx28nrHa0HLZ8HbkhpIpwlm8BUDko4nNZa1CMp8B5IeRxv6kh7AGg2YmSQNZPCawm6p+q/B4bSMP68zeSQs+KCfSMtsplXBX9ADZlZIeixh5WAWX5kBT0nalrVSX+B85TOflrp8XkO+rHdmNAVJ5xMcS6dKzUymFcGFs0ottSQpSxcAizUX/VJTEw6r7oLleWWs1KelUk3lWkJJKbn4ndeQP653PiBFSZmb3QQf9qG0zAZaYWZjlYDP4GxsWMsnCJm5AOjuE8FFvz4tjT1rSnrn7+ptkk4lOJZ2TGl57vwywbFU5XadfLAEq6Sli2epujkhz4JQf15L10T6tHQhLTO/ZIPyzMe1NP573D2LdfnMCv7ezmnp720W/yWYuvHnNuQBAPkt1wAARMgDQK4R8gCQY4Q8AOQYIQ8AOUbIA0CO5bV3DdCWyt2ieueOxa+5e7n5q4D0YiYPBMysGNwxWgn145WAD9odZO3uXYCQByrcvdLCYFBLPfFrm6t9QtlsFYAOR8gDgZqGXkV3r99sJpMtkwFq8sA7hs2sKOlEUJrJ3M5cQD161wAhBI2nbhMXYpExhDwA5Bg1eQDIMUIeAHKMkAeAHCPkASDHCHkAyDFCHgByjJAHgBwj5AEgx/4f4sinXeX1sA8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cfae21e748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The SVM with C = 0.1 has 16 support vectors and 2 margin defining vectors.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAERCAYAAACepNcKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XFsnOWdJ/DvL844ZtOFiUPuuJR244GjIgjtxE50Jks3JbW3cq1IxJkASVmSi1n7xDnb0kDgZLK7EqCFgEkpkVYOBCVsmxDiOFQ9arU2dq0WYqmxM6c2IKgYZ1nUrRpwxtw5OJ7Ev/tj3ncyM54Zz4zn9TvvO9+PhOz3nfH7PjaT7zzze5/3eURVQURE7rTA7gYQEZF1GPJERC7GkCcicjGGPBGRizHkiYhcjCFPeRGRZ0Vk2PgvYOzrFZHqpOftFpE643s1v497vFNEemc5l09Ens2znXUi0hnXlk5j3+58jpfNeZL2Z91242/aKyLH59o+41gZj5OqveQ+DHnKmRHUXlWtUdUaAH3GQ50A7k16er2qmo+HAGxOetxnXUsBVe1T1VZjc42qthr79lp1HvNNLxdG4H6kqvWquhnAgXzbYr7RGsdZKiK+pMe9xvnuyfcc5BwMecqHF0DY3FDVsPG1C0As4IxwCcf9XBhAZdzj1QBGrG5sHO88nSf5jS4jEfECqFPVWLCbf9M81QEwPx39xtiOUdWw8YYUmsM5yCEY8pQzI8zrjFJNS9LDI3ElmwCivft4v4l7/F4Ax1KdwyzjxJdyjB7ocWN/i7GvztjXKSLVKbZ9RuliN4DVcfufNX7ePJ5ZcvIZzzmeogfcG9eOj4zvq43jm+d51vjbdBrhDQDVxvZw3L54q3H101AhLAUwZnwfBnBTAY9NDrPQ7gaQM6lqjVG2eUxEauJKIscQDe8RREs1yWWRLgCtxuM+pOhNGgH+UVz5w2ecMwyj3CMiw4iWNDYD+GdVHTH2dyZtmz+7V0TqVbXVfJMxgr9XVQ+IyHGjbUC0jFGVojfda/zOPgB9Zg8ccW9UqvqYiFTHtb3S2N9qvAHkFejG71GX6rH4TwBEyRjylDej1t5nBK65r8vo0f4zUgS4qobMHjaipYRUajDzEwCAWDAvxdVa/rMAnjUCtzXFdiY3AagUkRoA3rhe9htpyiXmG5TXaN89MN7Iknv9Scy/w2dIXTI6jTS/r0lVQ8i+Tv8ZrpbFvMY2lSiGPOXM6AmHVDWcJtxGEA3b42kOYT7+WJrHhxHttSbU642ADxmhGgBi4bfZ6GG3qupj8dvIHJ7DAMaM8pN5jkokXkeIMd6gfEYbRkSkNd1zc2H8HftEpMXslYuIN/6NJseefB+in6b6ANRjljcQcjeGPOXruFmKwMwRM50AjseVcJIdMx4PpapRG+UTc2hlGFffDPoAvCwi9eZzjeA3tx9L3s70C5hlGjOsjdEosxkD8JHxfSWuXuBMYLR9tk8S8W1pNYdQGrtGENf+XHryxhvQvUYJKhRXuupV1Xrj+04APuPrs8bxyYWEs1ASEbkXR9cQEbkYQ56IyMUY8kRELsaQJyJyMYY8EZGL2T6E0ltZqcu//GW7m0FE5Cjv/+53n6rqstmeZ3vIL//yl/Ham2/a3QwiIkdZc/PN/5bN81iuISJyMYY8EZGLMeSJiFyMIU9E5GIMeSIiF2PIExG5GEOeiMjFGPJERC7GkCcicjHb73il7AyHQugJBhGemIB38WI0+P2o8WVaVpSIiCHvCMOhELqGhhC5cgUAEJ6YQNfQEAAw6IkoI5ZrHKAnGIwFvCly5Qp6gkGbWkRETsGQd4DwxERO+4mITAx5B/AuXpzTfiIiE0PeARr8fnjKyhL2ecrK0OD329QiInIKSy68ikg1AB8AqGqXFecoJebFVY6uIaJcWTW6plVVW0Vkt4hUq+qIRecpGTU+H0OdiHJW8JAXkQCAjwBAVfcW+vhERJQ9K2ryawAsFZFqEdltwfGJiChLVl14/cws0Rg9+wQi0iIip0Xk9IWxMYuaQEREVoT8RwBCxvchRHv2CVT1gKquVtXVSyorLWgCEREB1oR8H4yRNcbX31hwDiIiykLBQ15VQwDCZpmGQyiJiOxjyRBKVT1gxXGJiCg3vOOViMjFGPJERC7GkCcicjGGPBGRizHkiYhcjCFPRORiDHkiIhdjyBMRuRhDnojIxRjyREQuxpAnInIxhjwRkYsx5ImIXIwhT0TkYgx5IiIXY8gTEbmYJYuGEJEzDYdC6AkGEZ6YgHfxYjT4/ajx+Wb/QSpaDHkiAhAN+K6hIUSuXAEAhCcm0DU0BAAMegdjuYaIAAA9wWAs4E2RK1fQEwza1CIqBIY8EQGI9txz2U/OwJAnIgCAd/HinPaTM1gS8iLyrPG1xYrjE1HhNfj98JSVJezzlJWhwe+3qUVUCFb15FtE5CMAIYuOT0QFVuPzIVBbG+u5excvRqC2lhddHc6q0TWbVbXPomMTkUVqfD6GustYFfLVIgIA1aq616JzEBHRLCwp16jqXqMnv1RE6pIfF5EWETktIqcvjI1Z0QQiIoIFIS8iAREJGJufAZjx2U9VD6jqalVdvaSystBNICIigxXlmhCuXnC9CUCnBecgF+Ot9USFU/CQV9URoxwzBuAjVR0p9DnIvXhrPVFhWXLhVVUPWHFccr9Mt9Yz5Ilyxzteqajw1nqiwmLIU1HhrfVEhcWQp6LCW+uJCovzyVNRMevuHF1DVBgMeSo6vLWeqHBYriEicjGGPBGRizHkiYhcjCFPRORiDHkiIhdjyBMRuRhDnojIxRjyREQuxpAnInIxhjwRkYsx5ImIXIwhT0TkYgx5IiIX4yyUOeIi00TkJAz5HHCRaSJyGkvLNSKy28rjz7dMi0wTERUjy0JeROoArLHq+HbgItNE5DS88JoDLjJNRE5jSU1eRKpVtU9EWq04vl1uXb4cp37/+4R9XGQ6yfQ0yqamINPTgKrdrSEAEIEuWIAr5eXAAvbrSo1VF14rLTqubYZDIZwOhWbsX831SBOUTU3hP1VW4jqvFyJid3MIgKpiPBzGn8bGcKWiwu7m0Dwr+Nu62Yuf5TktInJaRE5fGBsrdBMskeqiKwC8/4c/2NCa4iXT0wz4IiMi0f8n09N2N4VsYMVnN5+IBEQkYHxfnfwEVT2gqqtVdfWSSmd0+nnRNUuqRRHwe9rbsaGxEfdv3Yp9HR0YHR3FnvZ2S861s63NkuMWkoiwfFaiCh7yqtqlql2Ilmy8hT6+XXjR1Tl2trWhyufDT996Cz86cgT/vbnZkvOEw2HsbGvDyRMnLDk+USFYdhXG6K3fpKojVp1jPjX4/fCUlSXs40XXuXuvvx+dDzyA5xoa0PnAA3ivv39OxwuHw/hlfz92xAW712tNX8Pr9eKl/fuxoqrKkuMTFQLveM2SeXGVUxoUznv9/fj5iy/i8qVLAIDP//Qn/PzFFwEAK9evz+uYZ0ZG8I0MPxsOh9H20EMYHx/HxqYm7GhuxkB/Pw6+8gqWVFaiubkZFy5cSNj2r1qVV1uIigFDPgc1HElTUL86dCgW8KbLly7hV4cO5R3ys/F6vfjRkSMAgDvXrsWO5mZ0d3fjkUcfjYX5zra2hG0iJ2PIk20+P38+p/3ZWFVdjb+f5ULovo4OjI2N4dzoKADg+7t2YU97O8bHx/HD/ftnbFfFlWP2tLcjPD4OX1UVHt61K+92Es0X3hlBtrl22bKc9mfD6/XiG+vX49WDB2P7wuFw7Pt9HR1YUVWFJ59+GkuWLAEAVFVV4UdHjuD7u3bh1VdembEd78mnn8ZL+/cz4MkxGPJkm69v346FixYl7Fu4aBG+vn37nI770v79GA2FsKGxERsaG9Hx3HOxx+5avx4dzz+fMOxxX0cHNjQ24oWODmwKBGZsZ7KzrQ3nRkexs60No8YnA6JiImrz2NmVt9+ur735pq1toMJZODGB/3rLLVk//73+fvzq0CF8fv48rl22DF/fvt2yenyp+/2HH+Iyh/y6xpqbbx5W1dWzPY81ebLVyvXrGepEFmK5hojIxRjyREQuxpAnInIxhjwRkYsx5MmVrJ6F0gkzTxIBDHlyIStnoeTMk+Q0HEJJtho4exaHBwfx6eef4/prr8W2detw12235X08cxbKl/bvj+3zer24cOFCIZobm3nyzIgrJlelEsCQJ9sMnD2Ll3p6cOnyZQDA+c8/x0s9PQCQd9BzFkqiRAx5ss3hwcFYwJsuXb6Mw4ODc+rNZ8JZKKnUsCZPtvn0889z2p+NVdXV+OUsC4/s6+jAnvb2hFkon3/uOWxobMTo6OiMbSInY8iTba6/9tqc9mfD6lkoiZyGIU+22bZuHRYtTKwYLlq4ENvWrZvTca2ehZIzT5KTZDULpYhUAdgEoEtVz4lIk6p2F6IBnIXSXXKdhbLQo2soPc5C6S6FnoUyAKAPwGYR6QVQD6AgIU+l7a7bbmOoE1ko25DvU9UzAM6IyDcBcKFTIiIHyDbkw2aJRlXfFhFLG0VERIUxa8iLyLWqOgogdoVJVd+e5WfqjG/rVfWxuTWRiIjylc3omv8lIn4AEJFV5vfpiEg1ouHeB6Da2CYiIhtkE/KnAfiMHv0ZAJWZnqyqI3G9d5+qcpIPmld72ttx59q1uHPtWpzstnd8wIbGRgTPnEnYt6+jAwOz3LBFVCjZ1OR9AMIA9hpDKXsBzPoKFZHdAFrn1rzSMRwKoScYRHhiAt7Fi9Hg96PGx+vbuRro70d4fBy/fvddAIk3QlnpZHc3NjY1zdi/o7kZJ7q6EqZI6O/vx8O7ds3puE7F1/n8y6YnH1LVl1X1f6jqtwCEsjmwqu4F0Coi3uTHRKRFRE6LyOkLY2M5Ntl9hkMhdA0NITwxAQAIT0yga2gIw6Gs/tSuUKiL+eFwGN7rrotte70zXn6WONHVlXL/xqYmvHnyZGx7dHQU18W1L9/jOhFf5/aYNeRV9YSIrACiNXkAN2V6vojE1+FDAFpSHPOAqq5W1dVLKjNWf0pCTzCIyJUrCfsiV66gJxi0qUXza2FZGf5s0SIsLCub87E2NjVhYGAAd65dmzC1wahxhyoQLZec7O7G6Ogo7ly7Fjvb2nD7ypUYHR1NuQ8A7t+6FRsaG2PHMI93/9at2NnWhl8ODGBnW1vKTw5/6ffHSjZvdnej+cEHY4+ZxzXLSjvb2mJ36u5pb59x3EztOPLjH8fak1wiKgal/jq3S1ZDKFX1nPH1DIDZXj11AMw6vBfAb/JtXKkwezbZ7neThWVlKDemNjC/Xk4Kglz9+t13MdDfjxc6OnDmzJmEueWTrVixAi/t34+T3d149ZVXsOPBB2fsq6ysxKZAABubmqL7Dh7EXevX4+SJE/jd++/D6/Xi3Oho2vNsCgRiJZv4Us2+jg6s/+Y3saO5Gfdv3YoLFy7AV1WVcJxgMBjb3mdMs5CuHXueeKKoZ88s5de5nayYu+YAohdqAwCgqu75vGkRb5pbzdPtdwsRwSKPJ1aqSd6ei7vWr8dP33pr1sU9zE+SG5uaEDR6lMn7hoeHcZcxR71/1Sr0vx0dQbxx06asykFmySYcDmNFVVVsf2h0FP1vv42dbW0YHx/H0KlTsfOkMls7in32zFJ9ndut4CGvqmGjHNOlqrzwmoUGvx+epFKFp6wMDf6Mo1UdT1VxKRKBOX9S8nY+gmfOxEob6YIuFLffvCY00N8fC+DkfTU1NbHRMAP9/aipqQGAhNr/bP7S78eeJ55AU9xF1FWrVmFTIICX9u/HT996C7V33JFx1M1s7Sj22TNL9XVuNy4aUgTM0QWlOOrALM0s8ngwdfnynEs1APC33/lObLm/f/3xjwFEA/DMyEhsBknTuXPncP/WrTh37hz+989+hgsXLszY5/V6cf/Wreh4/nmsWLECPzpyJOUbyIbGRvxw/35UxfXWTc0PPoi//c53EkoxZpnm1YMHcd111+FHR47EavLmdvxxH961K2M79nV0oN94E3jyqafm/HcstFJ+ndspq1korcRZKK1jx3C1XGehNInInHrw+RgdHcULHR0JwZtqn1twFkp3KfQslOQw5nA1czSDOVwNQFH2nOzubBC5FUO+wMzec4XHg8lIxLaPo5mGqxVjyNuhKmkkS7p9RE7GkM/BbAFu9p4DtbVora9HZ2+vbb1nDlcjIoAhn7VsArwnGESgtja2fJ351Y7es3fx4pSBzuFqRKWFa7xmKVWAB2prE+7Wq/B40Fpfj4ry8uh2eXl02+OZ9/ZyuBoRAQz5rGUT4JORCDp7ezE5NRXdnpqKbkci897eGp8PgdraWM/du3gxArW1rq/Hf+maa2aMNTeHJeZjoL8/YdHvTDjjJBUjlmuyZAb4tnXrUFFejsmpKRweHEwI8Aa/P1bCaa2vx+HBwViJxw41Pp/rQz1ZVVUVuru7E+4cPTeHuz/vWr8+412o8TjjJBUj9uSzZAb44cFBAIgFeHz5w+w99wSDuO8HP4iVeEotaO10ndeL+JlNg2fOwD9PJSrOOEnFiD35LMVfXB04exaTkUjKAC/F3vNcXL58GcffeAP3bdmC148exeZ77sHChXN7WdbU1ETDfdUqnOjqwqZAIDYvTTgcRttDD2F8fBwbm5pw1/r1eKGjAxfGxvDk009jT3s7xsfHsaKqCr6qKtzd1BSbuOzv29qwwrhz1rwTNpk546R/1aqUM06Oj49jR3MzNjY1Jdx96/f7YzNOPvnUU7G7bM22vLR/f+xGrQtjY/h2YyN+9tZbWFJZiebm5qKdlIzsx5DPAQO88I6/8Qb+rrkZf9fcHNu3ZevWOR3TDGb/qlUYHR1NmBTM6/XGpgu4c+3ahFkcz4yMoKamJjZ9wMNGsMZ7af9+7Glvx5mRkZRlHM44ScWG5Rqy1X1btmTczkdVVRWCwSBGR0djk3jF29fRgT3t7bFetDmL45IlSwBEe/up1jkw3ywqKyvTrjjFGSep2DDkyVavHz2acTtffr8fe9rbcXfShcx9HR1YUVWFJ59+Ohbq8bNJ9vf3o+2hh/D9LC+WpsIZJ6mYMOTJVpvvuQcvHzyI/3vxIl4+eBCb77mnIMfdFAjg/wSDM2aEvGv9enQ8/3zaYZHnRkdx7tw57Glvz7uH3Pzggzh54kRCT90cebOhsRH3b92KHc3NCI2OxrZNZs/84V27cKKrC3euXYv+t9+eMUJnX0cHNjQ24gWjrEOUDmehpILKdxbKYnD/1q148umnUVVVhYH+fnR3d7tqHhvOQukunIWSsmLHdMTF6pFHH8ULHR2x7ea4i8FETsWQL2FOm47Yav5Vq1zVcycCWJMvaZmmIyYid2BPvoRlOx1xTnPkGys8FWIxbiocVQX4/6QksSdfwtJNOxy/fzgUwhvvvIMln36Ko9/9LpZ8+ineeOcdDIdCKX9WFyzAeDjMlZ6KiKpG/58s4D/3UmRJT15EWoxvb1LVx6w4B82dOR9PfMkmeTrinmAQX/3iCxzauxeH9u4FAGzfvTvtHPlXysvxp7ExnP/0U4BBXxxEoAsW4Ioxg6pdeJHfHgUPeRGpA9CnqiEROS4idaraV+jz0NzFz8eT7h9ehceDV595JhbwAPDqM89gy4svpj7oggW4UlFhabvJeXiR3z5W9OR9xn8HAISM76lIzTYfz2Qkgh2PP56wb8fjj2Py+uutbhq5CNcctk/Bi3SqekBVDxib1QBOF/ocNH8a/H58fM012L57N6anp7F99258fM01XGGKcsI1h+1j2egaEakG0KuqIykeawHQAgA3LF9uVROoAOJLOltefBGT11+Pe1hLpRxxzWH7WDmEsk5V96Z6wOjpHwCi0xpY2AYqAE6xTHOVzUV+soYlY6pEpMUMeONCLBGVsFJdc7gYWDW65lkReQxAJYDNhT4HETkPPxHao+AhbwyXXFLo4xIRUe54CxwRkYsx5ImIXIwhT0TkYgx5IiIXY8gTEbkYQ56IyMUY8kRELsaQJyJyMYY8EZGLcY3XIpbT2qpERCmwJ1+kzJV0Gvx+vP6978Vm8Uu3tioRUSrsyRepnmAQgdpabFu3DgBiX7mSDhHlgiFfpCo8HrTW11/dLi9Ha309Bs6etbFVc8cSVPa48DUVAss1RWoyEkFnby8mp6ai21NT0e1IxOaW5Y8lqOyZfytzNSVz4Wv+rShXDPkiZQbg4cFBAMDhwcFYQDpVqhJUoLYWPcGgzS0rPpkWvibKBcs1RSp+bdWBs2cxGYk4fiUdt5agrMCFr6lQGPJFzG0r6ZglqG3r1qGivByTU1M4PDjo6BKUVbjwNRUKyzU0b9xYgrJKg98PT1lZwj4ufE35YE+e5o0bS1BWif9bcXQNzQVDnuaV20pQVuLfigrBsnKNiFRbdWwiIsqOJSEvInUAXs7muZHJSfzHhx9a0QwiopJnScirah+AsWyeW+HxAACDnojIAkVRk1+5bBkA4L2koP8vt9xiR3OIiFyjKELeZIY9ALx3/nxC756BT0SUO1tCXkRaALQAwPIbbkj5HAY+EdHc2RLyqnoAwAEAuH3lSp3t+fGBD1wt65Ri2HMWRyLKhSUhLyIBAKtFJKCqXYU+fqnW8M2ZCQO1tWitr0dnby+6hoYAgEFPRClZEvJGsBc83JOVWkmHC4kQUa6K6sLrXGQKfMAdoc9ZHIkoV64J+XgzavhG6M9H0FtZM+csjkSUK1eGfLLkGr5VYW91zdycxREAWuvrY7M4Bmpr53xsIiqMYlu2sSRC3rRy2TJLa/dW18w5iyNRcTM7euaqXuayjYB9gyNKKuSB9MMxTfmE/ienTuGDkyfxlW9/2/KaOWcmJCpemZZtZMjbpOLjjxHs6YGnogIXL17EZ3/911h6220Asgv8T06dwm9few21mzbhseZmDH34IfwrVrBmTlSCinHZxpJeGSo0PIyhri74Gxrwvddfx5rGRvz7z3+Oio8/xsply/AfH36Y8F8qH5w8idpNm7Bu2zYAgH/FCgTPnQPAlY+ISk265RntXLaxpHvywZ4e1AYCsYA2vwZ7euCrqclqHP4ijwf1ra2x/RXl5ai95RbsPHgQ/z42xpo5UQkxB0fEl2zsXraxJEM+NDyMYE8Pln7lKwkBXV5RgfrWVpwdGED3U0/BU1GByOQk/A0NWFlTk3AMs5Z/KRJBb2cn1m3bhvKKCkxNTmLw8GHc+ItfYNvevfP6exGRvYpx2caSC/nQ8DDeeeMNfPHVr+KRH/4Qe//xH9H26KP4sy99CQDw4dAQwn/8I/5qyxbUt7ait7MTQ13Rm3d9cUFvjtT5z3fcgXePHwcA1Le2YvDwYQydOIHbH3hg/n85IrJdsQ2OKLmQD/b04IuvfhV7Dx3C3kOHAACLvV78z4cfBgCs8Pux9ZlnsML4eJVcwom3ctky4BvfwJf//M8R7OnB2YEBXLx4Ebc/8ABuvOOO+fuliIjSENVZJ4G01O0rV+qbr702b+f76fPP47tHj2LBgqvXnKenpyEiGX/uB/fdhw2PPDLr8d87fz5h2w3TKRBR8Vlz883Dqrp6tueV3OiayOQkHt+xI2Hfc//0T5ianAQATE1O4sOhoYTt3s5ORIzt2axctizhv2xG6BARWaVkQj40PIzup57CRDiMRaEQdm/fjunpaezevh3/75e/xODhwwCAwcOHceTxxxO2zWGW+TDDvuLjj/Hbf/kX/Lq9Hb94+GH87uTJgv1uRETplES5xhwPfyXppiTvDTcgMjmJ5bfeij+8/35sNE3ytr+hYUY9Pp/z1wYCCRdzv/Ktb2HpbbexpENEOcu2XFMSF16DPT0zAh6Ilm6annjCsvOaQzU9FRVpx+Mvve0218+DT0T2cW3ImwE7EQ6nfc5EOIw3/uEfcOniRQBA+TXXYM3dd8+p1x5//vjee7z48fhWzKVDRGRyZcinK8+kYgY8AEx98QXePXYMAOYc9Ml3016ORCAAyjye2A1TyRdzT33yCU5+8AE8ixYhcukS/tvy5QmPM/CJKFeuvPCarjyTDZ2eRrCnZ85t8Bi99fKKCgDAQo8HZR4PgNQXc0998gle++1v8TfV1Xj9e9/D31RXo+/cOYxfusSROkSUN1f25DOVaIBoWWbqiy8y/vy/PvIIFnu9eV90jRhDL5OnOzg7MIDI5CRqA4GE45784ANsSjEX/cmREdxx440AUi9xyN49EWXiyp78Yq837WN/tWULPIsWZXWciXAYQ11dCA0P59wGf0MDhrq6EoZivnP0KC6Oj6d84/AsWoTW+npUlJcDuDoXfbq2JvfuiYhScWXIZxrTPtvF2GRXIpG8yje+mhrUBgI4/ZOf4Af33Yd3jh4FEK37p3rjiFy6hM7eXkxOTQEAJqem0Nnbi8ilSxnPY/buWcYholQsCXkRCYhInYjstuL4s8lUXpkIhzP29NP9TL7tWFhejvAf/5iwP9Ubx8avfQ0nhoZweHAQQHQu+hNDQ9j4ta/Neh7eZUtE6RS8Ji8i1QCgqn0i4hORalUdKfR5MslUXlns9WL5rbfi96dOZX28XN8U4qV7g0jeb9bdT46MYODsWUQuXcIDt98e2x8/5n62G7SymQefiEqDFRde7wXQa3wfAlAHYF5D/vRPfpL2seW33orQ6dNZH6vM48l7SgMg+gaRKuhTvXHcceONsVCPl+6OWWD2oZ4ch09U2qwIeS+AsbjtpRacI6P4se/J/vD++1kPr5zL6BqTeQE2/py5vnHMtoJVLtjLJyottgyhFJEWAC0AsPyGG+b13NnU18s8nhlDHPNlHsO84JvPG4c55t4Uf8fsXKTr5TPsidzDipAPA6g0vvcC+Cz5Cap6AMABIDpBWaEbkG4cfPk118CzaFHKoBcRqGpBeu/JfDU1czpeujH32U5/nC1ztSv27oncw4qQPwbAnBnNB6DPgnNktObuu/HusWPQ6enYPlmwAGvuvhsAUpZPCtVzt4JZ8gHilhg0avSFxt49kbsUPORVdUREVotIHYDwfI+sAbIrkcylfDLf4n+fdHfMWoW9eyJnK4n55KmwuMQhkf04nzxZJt0IHYY9UfFhyNOcmIHPkg5RcWLIU0Fw/D1RcWLIU8HxLlui4sGQJ8uxl09kH4Y8zSutJ7VfAAADW0lEQVSOwyeaX66cT56cI3k+fCIqLPbkyXYcoUNkHYY8FY1MtXuAoU+UD4Y8FaUZtXv28onyYvu0BiJyHsC/WXiK6wF8auHxreTktgPObr+T2w6w/Xaar7b/haoum+1Jtoe81UTkdDbzOxQjJ7cdcHb7ndx2gO23U7G1naNriIhcjCFPRORipRDyB+xuwBw4ue2As9vv5LYDbL+diqrtrq/JO5mIVNux6Ao5n4jsVtW9dreD7OfanryIBESkTkR2292WfBgra71sdzvyISItxn/P2t2WfBivmzontx/AGrvbkQ/zby4iLXa3JR8iUm1kT+HX5syTK0NeRKoBQFX7AITNbScx2j5mdztyZQRMn7FYu8/YdgzjtVJv/P2rnfjacbgWEfkIQMjuhuSpVVW7EH3tF8Vrx5UhD+BeAGHj+xAARwWNw/lw9e8dMrYdQ1VHVPUxY9PntHKZUeLrs7sdc7BZVW9y4u9g9N4/AgBV3Vssrx233vHqRWIveKldDSk1Rg/eVA3gmF1tmQujzNdqdzvyUGl3A+aoWkQAoNqB1xTWALFPg3XF0n639uTJZsYLvbdYejO5Mv6BtoqI1+62ZMsFvXizB9wHYKnTSn2Gz8zXfLHU5d0a8mFc7dF4AXxmY1tKVdH0ZHJhXDgza6khAE66AOiLu+hXNDXhbCVdsPwMDiv1IVqqMa8lhFAkF7/dGvLHcPUF4gPguN6N8WJfXSy9gVyISIsZ8A7sjdUhsYPgmAuAqtplXPSrRLTtThPC1X+rNwE4bWNb8tGHxNz5jY1tiXHtOHljCFYI0YtnRXVzgpsZoX4c0WsilYheSHPMm6xRnrkH0fbXq6oT6/KOZfy7HUP0360TPwkWXftdG/JEROTecg0REYEhT0Tkagx5IiIXY8gTEbkYQ56IyMUY8kRELubWuWuI8mLeLYqrdyz+s6qGM/8UUfFiT57IICI+445RM9SPmQFvTHfgtLt3iRjyRCZVNacwqEF0Tvz4ydXuhTOnCqASx5AnMsRN6OVT1eTFZhw5ZTIRa/JEV9WJiA9Ar1GacdzKXETJOHcNURaMiaduAi/EksMw5ImIXIw1eSIiF2PIExG5GEOeiMjFGPJERC7GkCcicjGGPBGRizHkiYhcjCFPRORi/x84z+R2s6vOIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cfae21e828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The SVM with C = 1 has 8 support vectors and 2 margin defining vectors.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAERCAYAAACepNcKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X9sVOeZL/DvY2N+xFfJ4ISr3DTNxZNsolBFO9ggOWwqNtT0yvUiBTOQhWYTLs7iq8hsk9JCrhz2nxQ1kDiIBWkFCRFkUwhgIKsutbZ27KJugqVgM6stiUrFDLuK2qoQM86VwWDguX/MOcPMeDyeGZ/j82O+HynynDPjM68nw3feec573ldUFURE5E9lTjeAiIjsw5AnIvIxhjwRkY8x5ImIfIwhT0TkYwx5soSIbBORfuO/sLGvS0RqMh63SUTqjdtq3k65f4+IdE3wXEER2VZkO+tFZE9KW/YY+zYVc7x8nidjf95tN17TLhE5akX7srWH/G+a0w0g7zOCOqCqtcZ2wLhrD4BnAQykPHypqm43bkcBrATQnXJ/0M62qmp3yvMtVNWVxu3ucX5l0s8jImFV7Sjk941A7lfVzcZ2YIJfyXWsAIBtAFYBaCn2OORN7MmTFQIA4uaGqsaNnx0AwuZ+EQmmPs64XZVyfw3SPxDsVnRwFujZQh5shHK9qu4195mvaTFUNa6qLUh8qFKJYcjTpBlhXm+UatZn3D2QUrIJI9G7T/VZyv3PAjic7TnMMk5qKUdEAkYpo8t8XqNMctR4fE2W7aBRBtkEYEHK/m3G75vHM0tOQeMxR40PqdQ2daW044Jxu8Y4vvk824zXZk9Kb7zG2O4fp4e+ABZ/s6DSxXINWUJVa42yzWYRqTV6jkAitM2STWqpxtSBRAlhAIlSzZjephHgF8xjmmFr9G5XGvv6Aew1tn+qqgPG/j0Z2+bvbheRparaYn7IGMHfpap7ReSo0TYgUeaoztKb7jL+5iCAbrMHjpQPKlXdLCI1KW2vMva3GB8ARQW68XfUZ7sv9RsAEUOeLGPWoY3ANfd1GD3anyJLgKtq1OxhA/hsnEPXYuw3AADJYL4Xd2r52wBsMwK3Jct2Lg8DqBKRWgCBlF72kXHKJeYHVMBo3yoYH2SZvf4M5uvwFbKXjM5gnL/XpKpRJD7UiHJiyNOkGT3hqKrGxwm3ASTC9ug4hzDv3zzO/f1I9FrT6vVGwEeNUA0DyfBbafSwW4wTl8lt5A7PfgCDqSdJjZ531nq48QEVNNowICIt4z22EMbr2C0i681euYgEUj9o2JOnfDHkySpHzVIEjBJKij0AjqaUcDIdNu6PZqtRG+UTc2hlHHc+DLoBvCMiS83HGsFvbm/O3M71B5hlGjOsU0be5DII4IJxuwpA1uGfRtvzHtlilnNSzkEMIKX9hfbkjbJV0Pi5zfh9KgHCWSiJiPyLo2uIiHyMIU9E5GMMeSIiH2PIExH5GEOeiMjHHB9CGaiq0ge+8Q2nm0FE5Clf/OY3l1V1zkSPczzkH/jGN/D+Rx853QwiIk9Z+Mgj/5nP41iuISLyMYY8EZGPMeSJiHyMIU9E5GMMeSIiH2PIExH5GEOeiMjHGPJERD7GkCci8jHHr3il/PRHo+iMRBAfHkagshINoRBqg7mWESUiYsh7Qn80io6+PozeugUAiA8Po6OvDwAY9ESUE8s1HtAZiSQD3jR66xY6IxGHWkREXsGQ94D48HBB+4mITAx5DwhUVha0n4jIxJD3gIZQCBXl5Wn7KsrL0RAKOdQiIvIKW068ikgNgCAAqGqHHc9RSsyTqxxdQ0SFsmt0TYuqtojIJhGpUdUBm56nZNQGgwx1IiqY5SEvImEAFwBAVbdbfXwiIsqfHTX5hQDuFZEaEdlkw/GJiChPdp14/cos0Rg9+zQisl5EzojImSuDgzY1gYiI7Aj5CwCixu0oEj37NKq6V1UXqOqC2VVVNjSBiIgAe0K+G8bIGuPnZzY8BxER5cHykFfVKIC4WabhEEoiIufYMoRSVffacVwiIioMr3glIvIxhjwRkY8x5ImIfIwhT0TkYwx5IiIfY8gTEfkYQ56IyMcY8kREPsaQJyLyMYY8EZGPMeSJiHyMIU9E5GMMeSIiH2PIExH5GEOeiMjHGPJERD5my6IhRORN/dEoOiMRxIeHEaisREMohNpgcOJfJNdiyBMRgETAd/T1YfTWLQBAfHgYHX19AMCg9zCWa4gIANAZiSQD3jR66xY6IxGHWkRWYMgTEYBEz72Q/eQNDHkiAgAEKisL2k/eYEvIi8g24+d6O45PRNZrCIVQUV6etq+ivBwNoZBDLSIr2NWTXy8iFwBEbTo+EVmsNhhEuK4u2XMPVFYiXFfHk64eZ9fompWq2m3TsYnIJrXBIEPdZ+wK+RoRAYAaVd1u03MQEdEEbCnXqOp2oyd/r4jUZ94vIutF5IyInLkyOGhHE4iICDaEvIiERSRsbH4FYMx3P1Xdq6oLVHXB7Koqq5tAREQGO8o1Udw54fowgD02PAf5GC+tJ7KO5SGvqgNGOWYQwAVVHbD6Oci/eGk9kbVsOfGqqnvtOC75X65L6xnyRIXjFa/kKry0nshaDHlyFV5aT2Qthjy5Ci+tJ7IW55MnVzHr7hxdQ2QNhjy5Di+tJ7IOyzVERD7GkCci8jGGPBGRjzHkiYh8jCFPRORjDHkiIh9jyBMR+RhDnojIxxjyREQ+xpAnIvIxhjwRkY8x5ImIfIwhT0TkY5yFskBcZJqIvIQhXwAuMk1EXmNruUZENtl5/KmWa5FpIiI3si3kRaQewEK7ju8ELjJNRF7DE68F4CLTROQ1ttTkRaRGVbtFpMWO4zvl8QcewOnf/S5tHxeZznD7Nspv3IDcvg2oOt0aAgARaFkZbk2fDpSxX1dq7DrxWmXTcR3TH43iTDQ6Zv8CrkeapvzGDfz3qircEwhARJxuDgFQVQzF4/jT4CBuzZzpdHNoiln+sW724id4zHoROSMiZ64MDlrdBFtkO+kKAF/8/vcOtMa95PZtBrzLiEji/8nt2043hRxgx3e3oIiERSRs3K7JfICq7lXVBaq6YHaVNzr9POmaJ1VXBPyWtjYsa2zEc2vWYEd7O2KxGLa0tdnyXBtaW205rpVEhOWzEmV5yKtqh6p2IFGyCVh9fKfwpKt3bGhtRXUwiJ+fPIkPDh7E/25utuV54vE4NrS24sSxY7Ycn8gKtp2FMXrrD6vqgF3PMZUaQiFUlJen7eNJ18n7vKcHe55/Hm82NGDP88/j856eSR0vHo/jVz09WJcS7IGAPX2NQCCAXbt3Y251tS3HJ7ICr3jNk3lylVMaWOfznh78686duHn9OgDg6z/9Cf+6cycAYN6SJUUd8+zAAP4yx+/G43G0vvQShoaGsLypCeuam9Hb04N9776L2VVVaG5uxpUrV9K2Q/PnF9UWIjdgyBegliNpLPXr/fuTAW+6ef06fr1/f9EhP5FAIIAPDh4EADy1aBHWNTfj+PHj+NGPf5wM8w2trWnbRF7GkCfHfH3pUkH78zG/pgZ/N8GJ0B3t7RgcHMTFWAwA8MONG7GlrQ1DQ0P4h927x2xXp5RjtrS1IT40hGB1NV7ZuLHodhJNFV4ZQY65e86cgvbnIxAI4C+XLMF7+/Yl98Xj8eTtHe3tmFtdjde3bsXs2bMBANXV1fjg4EH8cONGvPfuu2O2U72+dSt27d7NgCfPYMiTY769di2mzZiRtm/ajBn49tq1kzrurt27EYtGsayxEcsaG9H+5pvJ+55esgTtb72VNuxxR3s7ljU24u32dqwIh8ds57KhtRUXYzFsaG1FzPhmQOQmog6PnZ33xBP6/kcfOdoGss604WH82aOP5v34z3t68Ov9+/H1pUu4e84cfHvtWtvq8aXud+fP4yaH/PrGwkce6VfVBRM9jjV5ctS8JUsY6kQ2YrmGiMjHGPJERD7GkCci8jGGPBGRjzHkyZfsnoXSCzNPEgEMefIhO2eh5MyT5DUcQkmO6j13DgdOncLlr7/GfXffjRcWL8bT3/pW0cczZ6HctXt3cl8gEMCVK1esaG5y5smzA76YXJVKAEOeHNN77hx2dXbi+s2bAIBLX3+NXZ2dAFB00HMWSqJ0DHlyzIFTp5IBb7p+8yYOnDo1qd58LpyFkkoNa/LkmMtff13Q/nzMr6nBryZYeGRHezu2tLWlzUL51ptvYlljI2Kx2JhtIi9jyJNj7rv77oL258PuWSiJvIYhT455YfFizJiWXjGcMW0aXli8eFLHtXsWSs48SV6S1yyUIlINYAWADlW9KCJNqnrcigZwFkp/KXQWSqtH19D4OAulv1g9C2UYQDeAlSLSBWApAEtCnkrb09/6FkOdyEb5hny3qp4FcFZEvgOAC50SEXlAviEfN0s0qvqxiNjaKCIissaEIS8id6tqDEDyDJOqfjzB79QbN5eq6ubJNZGIiIqVz+ia/ysiIQAQkfnm7fGISA0S4d4NoMbYJiIiB+QT8mcABI0e/VkAVbkerKoDKb33oKpykg+aUlva2vDUokV4atEinDju7PiAZY2NiJw9m7ZvR3s7eie4YIvIKvnU5IMA4gC2G0MpuwBM+A4VkU0AWibXvNLRH42iMxJBfHgYgcpKNIRCqA3y/Hahent6EB8awr99+imA9Auh7HTi+HEsb2oas39dczOOdXSkTZHQ09ODVzZunNRxvYrv86mXT08+qqrvqOr/UdX/BSCaz4FVdTuAFhEJZN4nIutF5IyInLkyOFhgk/2nPxpFR18f4sPDAID48DA6+vrQH83rpfYFq07mx+NxBO65J7kdCIx5+9niWEdH1v3Lm5rw0YkTye1YLIZ7UtpX7HG9iO9zZ0wY8qp6TETmAomaPICHcz1eRFLr8FEA67Mcc6+qLlDVBbOrclZ/SkJnJILRW7fS9o3euoXOSMShFk2taeXluGvGDEwrL5/0sZY3NaG3txdPLVqUNrVBzLhCFUiUS04cP45YLIanFi3ChtZWPDFvHmKxWNZ9APDcmjVY1tiYPIZ5vOfWrMGG1lb8qrcXG1pbs35z+PNQKFmy+ej4cTS/+GLyPvO4ZllpQ2tr8krdLW1tY46bqx0Hf/azZHsyS0RuUOrvc6fkNYRSVS8aP88CmOjdUw/ArMMHAHxWbONKhdmzyXe/n0wrL8d0Y2oD8+fNjCAo1L99+il6e3rwdns7zp49mza3fKa5c+di1+7dOHH8ON57912se/HFMfuqqqqwIhzG8qamxL59+/D0kiU4cewYfvPFFwgEArgYi437PCvC4WTJJrVUs6O9HUu+8x2sa27Gc2vW4MqVKwhWV6cdJxKJJLd3GNMsjNeOLa+95urZM0v5fe4kO+au2YvEidowAKiqf75v2iQwzqXm4+33CxHBjIqKZKkmc3synl6yBD8/eXLCxT3Mb5LLm5oQMXqUmfv6+/vxtDFHfWj+fPR8nBhBvHzFirzKQWbJJh6PY251dXJ/NBZDz8cfY0NrK4aGhtB3+nTyebKZqB1unz2zVN/nTrM85FU1bpRjOlSVJ17z0BAKoSKjVFFRXo6GUM7Rqp6nqrg+Ogpz/qTM7WJEzp5NljbGC7poyn7znFBvT08ygDP31dbWJkfD9Pb0oLa2FgDSav8T+fNQCFteew1NKSdR58+fjxXhMHbt3o2fnzyJuiefzDnqZqJ2uH32zFJ9nzuNi4a4gDm6oBRHHZilmRkVFbhx8+akSzUA8Dff/35yub9/+tnPACQC8OzAQHIGSdPFixfx3Jo1uHjxIv7lF7/AlStXxuwLBAJ4bs0atL/1FubOnYsPDh7M+gGyrLER/7B7N6pTeuum5hdfxN98//tppRizTPPevn2455578MHBg8mavLmdetxXNm7M2Y4d7e3oMT4EXv/JTyb9OlqtlN/nTsprFko7cRZK+zgxXK3QWShNIjKpHnwxYrEY3m5vTwvebPv8grNQ+ovVs1CSx5jD1czRDOZwNQCu7Dk53dkg8iuGvMXccrFHruFqbgx5J1RnjGQZbx+RlzHkCzBRgLup98zhakQEcPm/vOVztZ6bLvbgcDUiAhjyecsnwN3Ue+ZwNSICGPJ5yyfA3dR7rg0GEa6rSz53oLIS4bo639fj/9usWWPGmpvDEovR29OTtuh3LpxxktyINfk8BSorswZ9aoA3hEJpNXnA2d5zbTDo+1DPVF1djePHj6ddOXpxEld/Pr1kSc6rUFNxxklyI/bk85RP+aNUe89uck8ggNSZTSNnzyI0RR+ynHGS3Ig9+Tzle7VeKfaeJ+PmzZs4euQI/nr1anx46BBWrlqFadMm97asra1NhPv8+TjW0YEV4XByXpp4PI7Wl17C0NAQljc14eklS/B2ezuuDA7i9a1bsaWtDUNDQ5hbXY1gdTWeaWpKTlz2d62tmGtcOWteCZvJnHEyNH9+1hknh4aGsK65GcubmtKuvg2FQskZJ1//yU+SV9mabdm1e3fyQq0rg4P4XmMjfnHyJGZXVaG5udm1k5KR8xjyBWCAW+/okSP42+Zm/G1zc3Lf6jVrJnVMM5hD8+cjFoulTQoWCASS0wU8tWhR2iyOZwcGUFtbm5w+4BUjWFPt2r0bW9racHZgIGsZhzNOktuwXEOO+uvVq3NuF6O6uhqRSASxWCw5iVeqHe3t2NLWluxFm7M4zp49G0Cit59tnQPzw6KqqmrcFac44yS5DUOeHPXhoUM5t4sVCoWwpa0Nz2ScyNzR3o651dV4fevWZKinzibZ09OD1pdewg/zPFmaDWecJDdhyJOjVq5ahXf27cP/u3oV7+zbh5WrVlly3BXhMP49EhkzI+TTS5ag/a23xh0WeTEWw8WLF7Glra3oHnLziy/ixLFjaT11c+TNssZGPLdmDdY1NyMaiyW3TWbP/JWNG3GsowNPLVqEno8/HjNCZ0d7O5Y1NuJto6xDNB7OQkmWKnYWSjd4bs0avL51K6qrq9Hb04Pjx4/7ah4bzkLpL5yFkvLilgnV3OBHP/4x3m5vT243p5wMJvIqhnwJc9OEam4Qmj/fVz13IoA1+ZLmpgnViMge7MmXsHwnVCuopGOs8GTFYtxkHVUF+P+kJLEnX8LymVAtnymWU2lZGYbica705CKqmvh/UsZ/7qXIlp68iKw3bj6sqpvteA6avHwmVCt0halb06fjT4ODuHT5MsCgdwcRaFkZbk2f7mgzeJLfGZaHvIjUA+hW1aiIHBWRelXttvp5aPLymY+n4Dnyy8pwa+ZMy9tK3saT/M6xoycfNP7bCyBq3CaXmmg+nnymWCaaCNccdo7lRTpV3auqe43NGgBnrH4OmjpcYYqs4KZV00qNbaNrRKQGQJeqDmS5bz2A9QBw/wMP2NUEskC+UywT5cJvhM6xcwhlvapuz3aH0dPfCySmNbCxDWQBTrFMk+W2VdNKiS1jqkRkvRnwxolYIiphXDXNOXaNrtkmIpsBVAFYafVzEJH38BuhMywPeWO45Gyrj0tERIXjJXBERD7GkCci8jGGPBGRjzHkiYh8jCFPRORjDHkiIh9jyBMR+RhDnojIxxjyREQ+xjVeXcxcSWdmRQVGRkc5+yMRFYw9eZfqj0Zx5JNPMPvyZRz6wQ8w+/JlHPnkk3HXViUiyoYh71KdkQgeunYN+7dvR1lZGfZv346Hrl1DZyTidNOIyENYrnGpmRUVeO+NN7B/+50p+d974w2s3rnTwVZNHhdzzh9fK7ICQ96lRkZHse7VV9P2rXv1VYzcd59DLZo8LuacP75WZBWWa1yqIRTCf82ahbWbNuH27dtYu2kT/mvWLE+vpJNrMWdKx9eKrMKevEulrq26eudOjNx3H1Z5/Os6F3POH18rsgpD3sX8tpIOF3POH18rsorj5ZrRkRH84fx5/OH8eaebQjZrCIVQUV6eto+LOWfH14qs4nhPfmZFBebNmYPPL11KBv3/ePRRh1tFdkgtQXHESG58rcgqoqqONuCJefP0o/ffT25/fulS2v0MfCKisRY+8ki/qi6Y6HG29eRFpEZVBwr9vXlz5iRvs3dPRDQ5ttTkRaQewDuTPc68OXOSoc+aPRFR4Wzpyatqt4gMWnW8zJq9ib17IqLcHD/xmq/UMg5wp5TDoCciGp9nQj6TGfqfs3dPRDQuR8bJi8h6ETkjImcGr1yZ1LHMuj1r90REYznSk1fVvQD2AokhlFYdN7N378dePRcSIaJC2DW6JgxggfFzyqX26v10Na05M2FDKIQPX34ZDaEQOvr6uJAIEY3LrtE1HQA67Dh2vsacqPVB774zEkG4rg4vLF4MAMmfnZEIe/NElJXjc9dMFT/07mdWVKBl6VLMnD49sT19emK7osLhlhGRW3l2dE0xpqJ3b2fNfGR0FHu6uvDC4sWYOX06Rm7cwIFTpzAyOmrJ8YnIf0oq5DNZPTGaWTMP19WhZelS7OnqsnQ1H7MGDwAtS5fiwKlTyecjIndw27KNJR3yQMqInIwraosJfLtr5qkzE/aeO4eR0VGE6+pYjydyCTcu21jyIW+azMRoX54+jd+eOIFvfu97aFm6NLnfrJn3njtnWTv9tpAIkZ/kWraRIe+QaH8/Ip2dqJg5E6MjIwg1NGBebS2A/Hr3X54+jf94/33UrViBzc3N6Dt/HqG5c1kzJypBbly2sWRG12QT7e/HJ0eO4PLs2fjBoUO4PHs2PjlyBNH+fgDZr6bNHJXz2xMnULdiBRa/8AIAIDR3LiIXLwJAsmbO1XyISsN4yzM6uWxjSYd8pLMT1x56CNv370dZWRm279+Paw89hEhn55jHZgv7P5w/jxkVFVja0oLpM2cCSJRo6h59FBv27UvW6FleISoNbly20XUrQ02F1BLNDw4dQlnZnc+627dvY+fq1RgdGcFwPI7KQAChhgYEjRJOqs8vXcJ//OM/YmFjIxa/8AKmz5yJGyMjOHXgAAZ++Ut8Z/v2qfyziMgFpmp0jeMrQ7lVtL8ffR0duGXUyV9dty7t/lfXrcOsP/4xuT0cj6OvI3HxbmbQz5szBzMbG/Hp0aMAgKUtLTh14AD6jh3DE88/b+efQUQu5bbBESUX8pHOzmTAA8CMaBSb1q7FG++9h1fXrcOMaBQQSfudW6OjiHR2Zu3Nm/sinZ0419uLq1ev4pvf/S7K773X3j+EiCgPJRfyw/F42naZCGbFYti5enWiB58R8OP9Xqpgbe2YDwAuakJEblByIV8ZCGQN7HhKiWa83ysEFzUhIjcomZA3T7bm6pGPp7yiAqGGhqKe1wz7aH8/Pjt5EhfuugvXR0fx2PLlePDJJ4s6JhFRvkpiCKV5sjVbwFcGAvizJ59M9tSzbdeFw1nr8YU+/8LGRrz84Yeo+e538e/79+PL06eLPiYRUT5KoiefebLVVBkIoOm112x73tRvD3+xenXyginz52dHjiRP0LKMQ0R28G3I51OeGY7HceTv/x7Xr14FAEyfNQsLn3lmUr321Oc3h2oG7r8fS1takvdNnzkTS1tacK63tySWLCQi5/gy5DPHwudiBjwA3Lh2DZ8ePgxg7Jj4QqV+e4j/8Y/o2rNnzAVToyMjycfPmzMHx774AnvPnkXlXXdh+OpVfPub38SSRYsm1Q4iKm2+rMmPV57Jh96+nXVag0JlfoP45NAhnDpwAAASF0x1dKSdzD395ZfovngRf7VwIT58+WX81cKF+GUshp5PP510W4iodPmyJz/RCJrps2bhxrVrOX//n370o5xTGkwk21DNTw4dwrneXoyOjIw5mXvit7/Fiixz0f/LZ5/hcQ7DJKIi+bInn2tM+1+sXo2KGTPyOo45pYE5K2UhQg0NKM+y9urVoaGsHxwVM2ZkXb+18q67ss6GSUSUD1+GfK4x7YWOlTenNChUsLYWdeEwZtx1V9r+G9euZf3gGL1+HXu6ujBy4wYAYOTGDezp6sLo9etpj8s17TERUSZbQl5EwiJSLyKb7Dj+RHKVV8yZJQtRzAVUZjumGT3zVNk+OJY/9hiO9fXhwKlTABJz0R/r68Pyxx4b8/vjTXtMRJTJ8pq8iNQAgKp2i0hQRGpUdcDq58klV3mlMhDAA48/jt8VcCFSoR8Kqcb7gMjc/+SDDwIATgwMoPfcOYxev47nn3giuT/XClbA2FWsANbuicieE6/PAugybkcB1AOY0pA/88//PO59Dzz+OKJnzuR9rMlMaQCMP1dOtg+OJx98MBnqqcwhoXXhMJa2tKBrz54x0x+nrlELcII0IkqwI+QDAAZTtqd8zt3Use+Zfv/FF3kPr5zM6BpTqKFhzJj9Qj84Ip2dqAuHx1wxO970x0Ai9Nm7JyJHhlCKyHoA6wHggfvvn9Lnzqe+Xl5RMen5akyp881PtNLUeCqMK2RNqVfM5sLePRHZEfJxAFXG7QCArzIfoKp7AewFEsv/Wd2A8cbBT581CxUzZmQNehGBqlrSe8+Ubb75QoyOjEx4xWw+OIUCUemxI+QPAzDXHQwC6LbhOXJa+Mwz+PTwYejt28l9UlaGhc88AwBZyydW9dztYJZ8gJQlBo0afTGylXIY+ET+ZHnIq+qAiCwQkXoA8akeWQPkVyKZTPlkqmUuMZjtitlCjSnlsHdP5Euianm1pCBPzJunH73/vqNtoITPL11K22bgE7nXwkce6VfVBRM9zpdz11Bx2Lsn8h9fTmtA1uAUCkTex5485ZQckcMTtUSexJCnvKSWclIDn2FP5G4MeSoYe/dE3sGQp6Kxd0/kfjzxSpbgoiZE7sSePFmKE6MRuQtDnizHidGI3IMhT7bLnBjNxNAnsp/j0xqIyCUA/2njU9wH4LKNx7eTl9sOeLv9Xm47wPY7aara/j9Vdc5ED3I85O0mImfymd/BjbzcdsDb7fdy2wG230luaztH1xAR+RhDnojIx0oh5Pc63YBJ8HLbAW+338ttB9h+J7mq7b6vyXuZiNQ4segKeZ+IbFLV7U63g5zn2568iIRFpF5ENjndlmIYK2u943Q7iiEi643/tjndlmIY75t6L7cfwEKn21EM8zUXkfVOt6UYIlJjZE9xa3PawJchLyI1AKCq3QDi5raXGG0fdLodhTICpttYrD1obHuG8V5Zarz+NV6+oowfAAACQElEQVR873jcehG5ACDqdEOK1KKqHUi8913x3vFlyAN4FkDcuB0F4Kmg8bgg7rzeUWPbM1R1QFU3G5tBr5XLjBJft9PtmISVqvqwF/8Go/d+AQBUdbtb3jt+veI1gPRe8L1ONaTUGD14Uw2Aw061ZTKMMl+L0+0oQpXTDZikGhEBgBoPnlNYCCS/Dda7pf1+7cmTw4w3epdbejOFMv6BtohIwOm25MsHvXizB9wN4F6vlfoMX5nvebfU5f0a8nHc6dEEAHzlYFtKlWt6MoUwTpyZtdQoAC+dAAymnPRzTU04XxknLL+Cx0p9SJRqzHMJUbjk5LdfQ/4w7rxBggA817sx3uwL3NIbKISIrDcD3oO9sXqkdxA8cwJQVTuMk35VSLTda6K482/1YQBnHGxLMbqRnjufOdiWJN+OkzeGYEWROHnmqosT/MwI9aNInBOpQuJEmmc+ZI3yzCok2r9UVb1Yl/cs49/tIBL/br34TdB17fdtyBMRkX/LNUREBIY8EZGvMeSJiHyMIU9E5GMMeSIiH2PIExH5mF/nriEqinm1KO5csfhTVY3n/i0i92JPnsggIkHjilEz1A+bAW9Md+C1q3eJGPJEJlU1pzCoRWJO/NTJ1Z6FN6cKoBLHkCcypEzoFVTVzMVmPDllMhFr8kR31ItIEECXUZrx3MpcRJk4dw1RHoyJpx4GT8SSxzDkiYh8jDV5IiIfY8gTEfkYQ56IyMcY8kREPsaQJyLyMYY8EZGPMeSJiHyMIU9E5GP/H4YZO5KXi6SRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cfae25ff60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The SVM with C = 100 has 4 support vectors and 2 margin defining vectors.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAERCAYAAACepNcKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X9sVNeZN/DvY2N+eZUMJqy6aZTiSbZVqKIdbJCApmJDjd51LaRgDCk0TXhx1l5FprspCWTlsLuvEtSFxEEp/gcSEElTUsCYrPpSa4NjynaTIAWbWakhaipm2GzVXRWwx7xy+GHgef/wvcPMeDxzZzyXO/fM9yNFnntnfO+xM3zn+DnnniuqCiIiMlOZ1w0gIiL3MOSJiAzGkCciMhhDnojIYAx5IiKDMeQpZyKyXUT6rf+arH3HRaQm5XWbRaTOeqz244Tnd4vI8SznCorI9jzbWSciuxPastvatzmf4zk5T8p+x223fqfHReRwIdqX2h7r+EnHTrePzMOQp5xYQR1Q1VpVrQXQaz21G8DjKS9frqr28xEAq1OeD7rXUkBVe1W11dpcqKqt1r4dbp3H/tDLhRXI51R1uaquBrAn37aISMA63pqEfTVWO1cDmG19+Izbl+85qbgx5ClXAQAxe0NVY9bXLgDxgLNCI5bwfTEAVQnP1wAYcLuxCQJ36DypH3QZiUgAQJ2qxoPd/p3mQ1Vj1gdOJGF3HQD7L6ZPrO10+8hADHnKiRXmdVappiXl6YGEkk0Txnr3iT5JeP5xAAfTncMu4ySWcqwe6mFrf4u1r87at1tEatJsB62SxGYACxL2b7e+3z6eXXIKWq85nNqztdtiteOc9bjGOr59nu3W72a3Fd4AUGNt9yfsS7QAt/8acstsAIPW4xiABybYRwaa4nUDyH9UtdYq22wRkdqEkshBjIX3AMZKNallkS4ArdbzQST3NgEAVoCfSyh/BK1zxmCVe0SkH2MljdUAfqyqA9b+3Snb9vfuEJHlqtpqf8hYwX9cVfeIyGGrbcBYmaM6TW/6uPUzBwH02j1wJHxQqeoWEalJaHuVtb/V+gDIK9CtnyNtTzvxLwCidBjylBer1t5rBa69r8vq0f4YaQJcVSN2DxtjJYJ0ajH+LwAA8WCejdu1/O0AtluB25pmO5MHAFSJSC2AQEIv+9AE5RL7AypgtW8NrA+yLPVs+/dwCelLRqcxwc9rU9UIJlGnt85tl8oC1jYm2EeGYbmGcmKFdMB6nC7cBjAWtocnOIT9fNcEz/cjTa/VCviIqm6BVWZQ1Yg1cLgdQGvqdpYfpR/AQWswdnlCsKeth1tBGwQQs/5SqJ3otbmwztubWPpKLetY5aCWdP85PE0vgOXW4+XWdrp9ZCD25Ckfh+1SBMbPmNkN4HBCCSfVQev5SLoatVU+sadWxgBssZ7qBfCGiNjBZAe/vb0ldTvTD2CXaUSkFWPBnfpzpDMI4Jz1uAq3By6TWG3P9iGT2JZWewqltWsACe3PtSdvla2C1tftqjogIo9bZalIQjlr3D4yj3AVSiIic7FcQ0RkMIY8EZHBGPJERAZjyBMRGYwhT0RkMM+nUAaqqvTer37V62YQEfnKZ7/5zUVVnZPtdZ6H/L1f/Srefu89r5tBROQrCx988D+dvI7lGiIigzHkiYgMxpAnIjIYQ56IyGAMeSIigzHkiYgMxpAnIjIYQ56IyGAMeSIig3l+xSs50x+JoCccRmxkBIHKStSHQqgNZrq1KBERQ94X+iMRdJ06hdGbNwEAsZERdJ06BQAMeiLKiOUaH+gJh+MBbxu9eRM94bBHLSIiv2DI+0BsZCSn/URENoa8DwQqK3PaT0RkY8j7QH0ohIry8qR9FeXlqA+FPGoREfmFKwOvIlIDIAgAqtrlxjlKiT24ytk1RJQrt2bXtKpqq4hsFpEaVR1w6TwlozYYZKgTUc4KHvIi0gTgHACo6o5CH5+IiJxzoya/EMBsEakRkc0uHJ+IiBxya+D1kl2isXr2SUSkRUROi8jpocFBl5pARERuhPw5ABHrcQRjPfskqrpHVReo6oJZVVUuNIGIiAB3Qr4X1swa6+snLpyDiIgcKHjIq2oEQMwu03AKJRGRd1yZQqmqe9w4LhER5YZXvBIRGYwhT0RkMIY8EZHBGPJERAZjyBMRGYwhT0RkMIY8EZHBGPJERAZjyBMRGYwhT0RkMIY8EZHBGPJERAZjyBMRGYwhT0RkMIY8EZHBGPJERAZz5aYhRORP/ZEIesJhxEZGEKisRH0ohNpgMPs3UtFiyBMRgLGA7zp1CqM3bwIAYiMj6Dp1CgAY9D7Gcg0RAQB6wuF4wNtGb95ETzjsUYuoEBjyRARgrOeey37yB4Y8EQEAApWVOe0nf3Al5EVku/W1xY3jE1Hh1YdCqCgvT9pXUV6O+lDIoxZRIbjVk28RkXMAIi4dn4gKrDYYRNOiRfGee6CyEk2LFnHQ1efcml2zWlV7XTo2EbmkNhhkqBvGrZCvEREAqFHVHS6dg4iIsnClXKOqO6ye/GwRqUt9XkRaROS0iJweGhx0owlERAQXQl5EmkSkydq8BGDc336qukdVF6jqgllVVYVuAhERWdwo10Rwe8D1AQC7XTgHGYyX1hMVTsFDXlUHrHLMIIBzqjpQ6HOQuXhpPVFhuTLwqqp73DgumS/TpfUMeaLc8YpXKiq8tJ6osBjyVFR4aT1RYTHkqajw0nqiwuJ68lRU7Lo7Z9cQFQZDnooOL60nKhyWa4iIDMaQJyIyGEOeiMhgDHkiIoMx5ImIDMaQJyIyGEOeiMhgDHkiIoMx5ImIDMaQJyIyGEOeiMhgDHkiIoMx5ImIDMZVKHPEm0wTkZ8w5HPAm0wTkd+4Wq4Rkc1uHv9Oy3STaSKiYuRayItIHYCFbh3fC7zJNBH5DQdec8CbTBOR37hSkxeRGlXtFZFWN47vlYfuvRcf/+53Sft4k+kUt26h/Pp1yK1bgKrXrSEAEIGWleHm1KlAGft1pcatgdcql47rmf5IBKcjkXH7F/B+pEnKr1/Hn1ZV4e5AACLidXMIgKpiOBbDHwcHcXP6dK+bQ3dYwT/W7V58lte0iMhpETk9NDhY6Ca4It2gKwB89oc/eNCa4iW3bjHgi4yIjP0/uXXL66aQB9z42y0oIk0i0mQ9rkl9garuUdUFqrpgVpU/Ov0cdHVItSgCfmt7O1Y0NOCJdeuws6MD0WgUW9vbXTnXxrY2V45bSCLC8lmJKnjIq2qXqnZhrGQTKPTxvcJBV//Y2NaG6mAQvzh2DO8cOID/3dzsynlisRg2trXh6JEjrhyfqBBcG4WxeusPqOqAW+e4k+pDIVSUlyft46Dr5J3t68PuJ5/EK/X12P3kkzjb1zep48ViMfyqrw8bEoI9EHCnrxEIBLCrsxNzq6tdOT5RIfCKV4fswVUuaVA4Z/v68K+vv44b164BAC7/8Y/419dfBwDMW7Ysr2OeGRjAX2b43lgshrZnnsHw8DBWNjZiQ3MzTvT1Ye+bb2JWVRWam5sxNDSUtB2aPz+vthAVA4Z8Dmo5k6agfr1/fzzgbTeuXcOv9+/PO+SzCQQCeOfAAQDAI0uWYENzM7q7u/Hc88/Hw3xjW1vSNpGfMeTJM5cvXMhpvxPza2rwwywDoTs7OjA4OIjz0SgA4EebNmFrezuGh4fxk87OcdvVCeWYre3tiA0PI1hdjWc3bcq7nUR3Cq+MIM/cNWdOTvudCAQC+Mtly7Bv7974vlgsFn+8s6MDc6ur8dK2bZg1axYAoLq6Gu8cOIAfbdqEfW++OW470UvbtmFXZycDnnyDIU+e+fb69ZgybVrSvinTpuHb69dP6ri7OjsRjUSwoqEBKxoa0PHKK/HnHl22DB2vvpo07XFnRwdWNDTgtY4OrGpqGredyca2NpyPRrGxrQ1R6y8DomIi6vHc2XkPP6xvv/eep22gwpkyMoI///rXHb/+bF8ffr1/Py5fuIC75szBt9evd60eX+p+9/nnuMEpv8ZY+OCD/aq6INvrWJMnT81btoyhTuQilmuIiAzGkCciMhhDnojIYAx5IiKDMeTJSG6vQumHlSeJAIY8GcjNVSi58iT5DadQkqdOfPop3jp5EhcvX8Y9d92Fp5YuxaPf/Gbex7NXodzV2RnfFwgEMDQ0VIjmxleePDNgxOKqVAIY8uSZE59+il09Pbh24wYA4MLly9jV0wMAeQc9V6EkSsaQJ8+8dfJkPOBt127cwFsnT06qN58JV6GkUsOaPHnm4uXLOe13Yn5NDX6V5cYjOzs6sLW9PWkVyldfeQUrGhoQjUbHbRP5GUOePHPPXXfltN8Jt1ehJPIbhjx55qmlSzFtSnLFcNqUKXhq6dJJHdftVSi58iT5iaNVKEWkGsAqAF2qel5EGlW1uxAN4CqUZsl1FcpCz66hiXEVSrMUehXKJgC9AFaLyHEAywEUJOSptD36zW8y1Ilc5DTke1X1DIAzIvIdALzRKRGRDzgN+ZhdolHVD0TE1UYREVFhZA15EblLVaMA4iNMqvpBlu+psx4uV9Utk2siERHly8nsmr8XkRAAiMh8+/FERKQGY+HeC6DG2iYiIg84CfnTAIJWj/4MgKpML1bVgYTee1BVucgH3VFb29vxyJIleGTJEhzt9nZ+wIqGBoTPnEnat7OjAyeyXLBFVChOavJBADEAO6yplMcBZH2HishmAK2Ta17p6I9E0BMOIzYygkBlJepDIdQGOb6dqxN9fYgND+PfP/oIQPKFUG462t2NlY2N4/ZvaG7Gka6upCUS+vr68OymTZM6rl/xfX7nOenJR1T1DVX9G1X9XwAiTg6sqjsAtIpIIPU5EWkRkdMicnpocDDHJpunPxJB16lTiI2MAABiIyPoOnUK/RFHv2ojFGowPxaLIXD33fHtQGDc288VR7q60u5f2diI944ejW9Ho1HcndC+fI/rR3yfeyNryKvqERGZC4zV5AE8kOn1IpJYh48AaElzzD2qukBVF8yqylj9KQk94TBGb95M2jd68yZ6wmGPWnRnTSkvx8xp0zClvHzSx1rZ2IgTJ07gkSVLkpY2iFpXqAJj5ZKj3d2IRqN4ZMkSbGxrw8Pz5iEajabdBwBPrFuHFQ0N8WPYx3ti3TpsbGvDr06cwMa2trR/OfxFKBQv2bzX3Y3mp5+OP2cf1y4rbWxri1+pu7W9fdxxM7XjwM9+Fm9PaomoGJT6+9wrjqZQqup56+sZANnePXUA7Dp8AMAn+TauVNg9G6f7TTKlvBxTraUN7K83UoIgV//+0Uc40deH1zo6cObMmaS15VPNnTsXuzo7cbS7G/vefBMbnn563L6qqiqsamrCysbGsX179+LRZctw9MgR/OazzxAIBHA+Gp3wPKuamuIlm8RSzc6ODiz7znewobkZT6xbh6GhIQSrq5OOEw6H49s7rWUWJmrH1hdfLOrVM0v5fe4lN9au2YOxgdomAFBVc/7edElggkvNJ9pvChHBtIqKeKkmdXsyHl22DL84dizrzT3svyRXNjYibPUoU/f19/fjUWuN+tD8+ej7YGwG8cpVqxyVg+ySTSwWw9zq6vj+SDSKvg8+wMa2NgwPD+PUxx/Hz5NOtnYU++qZpfo+91rBQ15VY1Y5pktVOfDqQH0ohIqUUkVFeTnqQxlnq/qequLa6Cjs9ZNSt/MRPnMmXtqYKOgiCfvtMaETfX3xAE7dV1tbG58Nc6KvD7W1tQCQVPvP5i9CIWx98UU0Jgyizp8/H6uamrCrsxO/OHYMixYvzjjrJls7in31zFJ9n3uNNw0pAvbsglKcdWCXZqZVVOD6jRuTLtUAwA++//347f5++rOfARgLwDMDA/EVJG3nz5/HE+vW4fz58/i/v/wlhoaGxu0LBAJ4Yt06dLz6KubOnYt3DhxI+wGyoqEBP+nsRHVCb93W/PTT+MH3v59UirHLNPv27sXdd9+Ndw4ciNfk7e3E4z67aVPGduzs6ECf9SHw0ssvT/r3WGil/D73kqNVKN3EVSjd48V0tVxXobSJyKR68PmIRqN4raMjKXjT7TMFV6E0S6FXoSSfsaer2bMZ7OlqAIqy5+R1Z4PIVAz5AiuWiz0yTVcrxpD3QnXKTJaJ9hH5GUM+B9kCvJh6z5yuRkQAb//nmJOr9YrpYg9OVyMigCHvmJMAL6beM6erERHAkHfMSYAXU++5NhhE06JF8XMHKivRtGiR8fX4P5kxY9xcc3taYj5O9PUl3fQ7E644ScWINXmHApWVaYM+McDrQ6Gkmjzgbe+5Nhg0PtRTVVdXo7u7O+nK0fOTuPrz0WXLMl6FmogrTlIxYk/eISflj1LtPReTuwMBJK5sGj5zBqE79CHLFSepGLEn75DTq/VKsfc8GTdu3MDhQ4fwvbVr8fN338XqNWswZcrk3pa1tbVj4T5/Po50dWFVU1N8XZpYLIa2Z57B8PAwVjY24tFly/BaRweGBgfx0rZt2NrejuHhYcytrkawuhqPNTbGFy77YVsb5lpXztpXwqayV5wMzZ+fdsXJ4eFhbGhuxsrGxqSrb0OhUHzFyZdefjl+la3dll2dnfELtYYGB/Hdhgb88tgxzKqqQnNzc9EuSkbeY8jngAFeeIcPHcJfNzfjr5ub4/vWrls3qWPawRyaPx/RaDRpUbBAIBBfLuCRJUuSVnE8MzCA2tra+PIBz1rBmmhXZye2trfjzMBA2jIOV5ykYsNyDXnqe2vXZtzOR3V1NcLhMKLRaHwRr0Q7Ozqwtb093ou2V3GcNWsWgLHefrr7HNgfFlVVVRPecYorTlKxYciTp37+7rsZt/MVCoWwtb0dj6UMZO7s6MDc6mq8tG1bPNQTV5Ps6+tD2zPP4EcOB0vT4YqTVEwY8uSp1WvW4I29e/H/vvwSb+zdi9Vr1hTkuKuamvAf4fC4FSEfXbYMHa++OuG0yPPRKM6fP4+t7e1595Cbn34aR48cSeqp2zNvVjQ04Il167ChuRmRaDS+bbN75s9u2oQjXV14ZMkS9H3wwbgZOjs7OrCioQGvWWUdoolwFUoqqHxXoSwGT6xbh5e2bUN1dTVO9PWhu7vbqHVsuAqlWbgKJTlSLAuqFYPnnn8er3V0xLebEwaDifyKIV/CimlBtWIQmj/fqJ47EcCafEkrpgXViMgd7MmXMKcLquVU0rHu8FSIm3FT4agqwP8nJYk9+RLmZEE1J0ssJ9KyMgzHYrzTUxFR1bH/J2X8516KXOnJi0iL9fABVd3ixjlo8pwsqJbrHaZuTp2KPw4O4sLFiwCDvjiIQMvKcHPqVE+bwUF+bxQ85EWkDkCvqkZE5LCI1Klqb6HPQ5PnZD2enNfILyvDzenTC95W8jcO8nvHjZ580PpvD4CI9ZiKVLb1eJwssUyUDe857J2CF+lUdY+q7rE2awCcLvQ56M7hHaaoEIrprmmlxrXZNSJSA+C4qg6kea4FQAsAfOXee91qAhWA0yWWiTLhX4TecXMKZZ2q7kj3hNXT3wOMLWvgYhuoALjEMk1Wsd01rZS4MqdKRFrsgLcGYomohPGuad5xa3bNdhHZAqAKwOpCn4OI/Id/EXqj4CFvTZecVejjEhFR7ngJHBGRwRjyREQGY8gTERmMIU9EZDCGPBGRwTwP+dGrV71uAhGRsTwPeQD4788/x39//rnXzSAiMo7nIT+9ogLz5swBAAY9EVGBFc3t/+ygP2sF/Z99/eteNoeIyAhFE/K2eXPm4OyFC/FefSmHPe+kQ0ST5Xm5Jp15c+aUfAkn13urEhGlU5Qhb0sM+lIL+0x30iEicqroyjWp4rV6q4RTKuUbU++kwxKUc/xdUSEUfcjbSm1g1sQ76fBmzs7xd0WFUtTlmnRKpVZv4r1VWYJyjr8rKhTf9OQTlUKv3sR7q5pagnIDf1dUKL4MeZvp0y1Nu5OOiSUot/B3RYXiu3JNKk639A8TS1Bu4e+KCsXXPflEpvfqTWBiCcot/F1RoYiqetqAh+fN0/fefrugxzx74QIABj0RmWvhgw/2q+qCbK9zrScvIjWqOuDW8TMphYFZIiInXKnJi0gdgDfcOHYuWKsnolLnSk9eVXtFZNCNY+eKvXoiKmW+n13jVCmvg0NEpcuY2TVOpK6DY2PvnohM5UnIi0gLgBYAuPcrX7nj57fDHgCnXRKR0TwJeVXdA2APMDaF0os22Py2yiVXJiSiXLg1u6YJwALrqy/4oWbPG4kQUa5cCXlV7VLVWara5cbx3VLsSyRwZUIiylVJDbw6VazTLrkyIRHliiGfQT7r4dg18+kVFbg6OlrQmjlXJiSiXJXMPPl85VLC6Y9EcOjDDzHr4kW8+7d/i1kXL+LQhx8WrGbOlQmJil9/JIKXu7vx3E9/ipe7uz0fM2PIO+RkYLYnHMb9V65g/44dKCsrw/4dO3D/lSsFq5nXBoNoWrQo3nMPVFaiadEizq4hKhLFODnCyFUo3Za6yuXvP/4Yvz16FL//7nfxkw0bUFZ2+7Pz1q1bWPv663huxQpP2kpEd87L3d0TllRfbGws6Lk8X4XSLyL9/Qj39GAkFkNlIIBQfT2CtbUZvyexVn/p00/xX++/j0WrVmFLczP+cceOpNdueOEFXL3nHjd/BCIqEsU4OaKkQz7S349TXV24OToKABiJxXCqa2zWp5OgB4CD//ZvWLRqFZY+9RQA4Pm2NgDAPz3/PDa88AK+mDEDa1gzJyoJxTg5oqRr8uGennjA226OjiLc0+P4GDNnzsTy1lZMnT4dAPAnM2fi/2zejB/u24ehe+7Bmm99izVzohJRjJMjSrInn1iiSWckFkP3yy87KuGMXr2K47t3Y+lTT2Hq9Om4fvUqTr71Fu57/308lVK6ISKzFeNtG0su5FNLNBOxPwCylXBC9fXx55e3tuLkW2/ho8OH8bW/+qsCt5yI/KA2GCyqv95LLuTTlWiysUs46ULe3hfu6cGnJ05g9OpVLFm9Glfvv5+rWxKR50ou5Ccq0Uzm+4K1tROWc/yyuiURmankQr4yEMgr6CsDgbzOV6zr4BBRaSiZkM822JpJeUUFQvX1kzr/9C++wCfHjqH/8mXMmD0b31i5EvctXjypYxIRZVMSUyjtwdZ0AV8ZCODPFy+O99TTbS9qaso6b97J+a9fvgwAuHLpEv5j/378/uOP8z4mEZETJdGTn2iwtTIQQOOLL7p23kx/PeiNGzh76BDKZ89mCYeIXGNsyDspz4zEYjj0D/+Aa19+CQCYOmMGFj722KR67YnnzzZV0+7Zc2CWiNxiZMg7nQsPIB7wAHD9yhV8dPAggOzLGmTjZKpmZSCQtA7OZxcv4qP/+R9X1qInotJkZE0+n7nwNr11K6dlDSaSbYA3cTB33pw5GL52De9Ho6gPhfDzv/s71IdCni9RSkT+Z2RPPlvATp0xA9evXMn4/T997jnHq1Kmk2mqZrrjHv3tb7F68WI8tXQpAMS/9oTD7M0TUd6M7MlnmtP+rbVrUTFtmqPj2EsaRPr7c25DqL4e5RUV4/ZPnTEj7QdHxbRpaF2+HNOnTgUATJ86dWw7zTGIiJwyMuQzzWnPda58rqtS2oK1tVjU1IRpM2cm7b9+5UraD47Ra9ew+/hxXL1+HQBw9fp17D5+HCNffpn1toNERBNxJeRFpElE6kRksxvHzyZTecVeWTIX+S6FEKytxRSrZ54o3QfHym98A0dOncJbJ08CAN46eRJHTp3C4/PmAch820EiookUvCYvIjUAoKq9IhIUkRpVHSj0eTLJVF6pDARw70MP4Xc5XIiU75IGwMQfEKn7F993HwDg6MAATnz6KUavXcOTDz8c3x/p78cnx47h3MyZuDY6yitmicgRNwZeHwdw3HocAVAH4I6G/Ol/+ZcJn7v3oYcQOX3a8bEmu6TBRAOw6T44Ft93XzzUE0X6+/HhoUO4cv/9+Pt9+/DChg0Y2LcPABj0RJSRG+WaAIDBhO3ZLpwjo8S576n+8NlnjqdXFmJJg3QDsLl+cIR7enDl/vuxY/9+lJWVYcf+/bj2ta/h7KFDebeLiEqDJwOvItIiIqdF5PTg0NAdPbeT+np5RQW+tXYtGl98cdIXRdkDsJNZC6di+nT8s9Vzt/3zvn2YOXMma/VElJEb5ZoYgCrrcQDApdQXqOoeAHsA4OF587TQDZhoHvzUGTNQMW1a2qAXEajqpObGTyTTevNOjF69ihc2bEja98KGDbjn6tWkK2a5NAIRpXKjJ38QgH31ThBArwvnyGjhY49BypJ/NCkrw8LHHpuwfLLke9/DD159tSC990IL1ddjxhdfYPP69bh16xY2r1+PGV98kXTF7Lw5c9irJ6JxCt6TV9UBEVkgInUAYnd6Zg2QfEu+iW7Gnem5YpP487y+di3uuXoVoTVrxrU5sVcP8AYlRASIasGrJTl5eN48fe/ttz1tg4nOXrgAgEFPZKqFDz7Yr6oLsr3OyCte6fZtB1nCISptDHmD2bV6AAx6ohJl5CqUlCz1ZuI2lnKIzMeefAmxe/bs3ROVDvbkS1Rq7569eiIzsSdf4jhAS2Q2hjyxhENkMIY8xbFXT2QehjwlYa+eyCwceKW0ODBLZAbPlzUQkQsA/tPFU9wD4KKLx3eTn9sO+Lv9fm47wPZ76U61/WuqOifbizwPebeJyGkn6zsUIz+3HfB3+/3cdoDt91KxtZ01eSIigzHkiYgMVgohv8frBkyCn9sO+Lv9fm47wPZ7qajabnxN3s9EpMaLm66Q/4nIZlXd4XU7yHvG9uRFpElE6kRks9dtyYd1Z603vG5HPqwbtbeIyHav25IP631T5+f2A1jodTvyYf/ORaTF67bkQ0RqrOxp8rotNiNDXkRqAEBVewHE7G0/sdo+6HU7cmUFTK91s/agte0b1ntlufX7r/Hje8fnWkTkHICI1w3JU6uqdmHsvV8U7x0jQx7A4wBi1uMIAF8Fjc8Fcfv3HcHtm7r7gqoOqOoWazPot3KZVeLr9bodk7BaVR/w489g9d7PAYCq7iiW946pV7wGkNwLnu1VQ0qN1YO31QA46FVbJsMq87V63Y48VHndgEmqEREAqPHhmMJCIP7XYF2xtN/Unjx5zHqjHy+W3kyurH+grSKWOZypAAABoUlEQVQS8LotThnQi7d7wL0AZvut1Ge5ZL/ni6Uub2rIx3C7RxMAcMnDtpSqounJ5MIaOLNrqREAfhoADCYM+hVNTdiplAHLS/BZqQ9jpRp7LCGCIhn8NjXkD+L2GyQIwHe9G+vNvqBYegO5EJEWO+B92BurQ3IHwTcDgKraZQ36VWGs7X4Twe1/qw8AOO1hW/LRi+Tc+cTDtsQZO0/emoIVwdjgWVFdnGAyK9QPY2xMpApjA2m++ZC1yjNrMNb+5arqx7q8b1n/bgcx9u/Wj38JFl37jQ15IiIyt1xDRERgyBMRGY0hT0RkMIY8EZHBGPJERAZjyBMRGczUtWuI8mJfLYrbVyz+WFVjmb+LqHixJ09kEZGgdcWoHeoH7YC3ljvw29W7RAx5Ipuq2ksY1GJsTfzExdUehz+XCqASx5AnsiQs6BVU1dSbzfhyyWQi1uSJbqsTkSCA41Zpxnd35iJKxbVriBywFp56AByIJZ9hyBMRGYw1eSIigzHkiYgMxpAnIjIYQ56IyGAMeSIigzHkiYgMxpAnIjIYQ56IyGD/H02qDVKrivXlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cfac442128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_contour_2d_classification(X, y, lls_classifier, delta, \"Linear classifier\")\n",
    "\n",
    "print_important_vectors(svm_smallC, \"SVM with C = \" + str(small_C))\n",
    "plot_contour_2d_classification(X, y, svm_smallC_classifier, 2 * delta, \"SVM classifier with C = \" + str(small_C),\n",
    "                               lambda: add_important_vectors(svm_smallC))\n",
    "\n",
    "print_important_vectors(svm_mediumC, \"SVM with C = \" + str(medium_C))\n",
    "plot_contour_2d_classification(X, y, svm_mediumC_classifier, 2 * delta, \"SVM classifier with C = \" + str(medium_C), \n",
    "                               lambda: add_important_vectors(svm_mediumC))\n",
    "\n",
    "print_important_vectors(svm_bigC, \"SVM with C = \" + str(big_C))\n",
    "plot_contour_2d_classification(X, y, svm_bigC_classifier, 2 * delta, \"SVM classifier with C = \" + str(big_C),\n",
    "                               lambda: add_important_vectors(svm_bigC))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In most cases, the support vector classifiers perform better on the training set than the lls classifier. As we can see (most of the times; sometimes skewed data sets distort these properties) the number of support vectors decreases with increasing *C*, while the number of margin defining vectors increases when *C* increases, but at a far slower rate than the increase of *C*. Furthermore, we can see that, with increasing value of *C*, the hyperplane of the support vector classifiers rotates to label as many training examples correctly as possible, which sometimes leads to the hyperplane splitting the two dimensional plane in a way that separates the training examples, but does not reflect the underlying exponential distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.4\n",
    "However, one cannot confidently predict how well a model generalizes by only testing it on the set it was trained on. So now, we create a bigger test set containing examples from the same distribution and test our algorithms on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The lls classifier has accuracy 0.8085\n",
      "The svm classifier with C = 0.1 has accuracy 0.919\n",
      "The svm classifier with C = 1 has accuracy 0.922\n",
      "The svm classifier with C = 100 has accuracy 0.9255\n"
     ]
    }
   ],
   "source": [
    "big_N = 1000\n",
    "big_set = create_dataset(big_N, exp_params)\n",
    "big_X = big_set[:, :(-1)]\n",
    "big_y = big_set[:, -1]\n",
    "\n",
    "print(\"The lls classifier has accuracy \" + str(accuracy(big_y, lls_classifier(big_X))))\n",
    "for (C, svm) in [(small_C, svm_smallC_classifier), (medium_C, svm_mediumC_classifier), (big_C, svm_bigC_classifier)]:\n",
    "    print(\"The svm classifier with C = \" + str(C) +  \" has accuracy \" + str(accuracy(big_y, svm(big_X))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As one might have predicted, after seeing the steep hyperplane learned by the support vector classifier with the highest regularization parameter, this classifier performs poorer on the test set, because its bias from the training set is too high. However, most of the times, all support vector classifiers perform better than the linear classifier, achieving accuracies around 90%, while the linear classifier often caps at 80%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.6\n",
    "The support vector classifier can also be used to classify non-linear data. To show this, we first create a data set containing 100 2 dimensional vectors, 50 of which, labelled -1, have a length smaller than 1 and the other 50, labelled 1, have a length between 1 and 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_circle_set(N):\n",
    "    \"\"\"Creates a set containg N random 2 dimensional vectors of length smaller 1 and N vectors of length between 1 and 2\n",
    "    :param N: Number of vectors to generate for each length range\n",
    "    \n",
    "    :returns: ndarray. (2N, 3) matrix with first N entries labelled -1 and second N labelled 1\n",
    "    \"\"\"\n",
    "    X = np.zeros((2*N, 2))\n",
    "    for i in range(N):\n",
    "        X[i, :] = np.random.uniform(0, 1, 2)\n",
    "        while sum(X[i,:]**2) >= 1:\n",
    "            X[i, :] = np.random.uniform(0, 1, 2)\n",
    "    for j in range(N, 2*N):\n",
    "        X[j, :] = np.random.uniform(0, 2, 2)\n",
    "        while sum(X[j,:]**2) <= 1 or sum(X[j, :]**2) >= 2:\n",
    "            X[j, :] = np.random.uniform(0, 2, 2)\n",
    "        \n",
    "    X = np.random.choice([-1, 1], (2*N, 2)) * X    \n",
    "    \n",
    "    return np.concatenate((X, np.concatenate((-1 * np.ones((N, 1)), np.ones((N, 1))), axis = 0)), axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "circle_N = 50\n",
    "circle_set = create_circle_set(circle_N)\n",
    "circle_X = circle_set[:, :(-1)]\n",
    "circle_y = circle_set[:, -1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we consider feature maps, we take a look at how a support vector classifier with the standard kernel performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "circle_C = 10\n",
    "circle_svm_standard = SMO(circle_X, circle_y, circle_C, max_iter) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAERCAYAAACO6FuTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X9sHeWZL/DvE+cHbrr0EEMFbnYvOWFjkcqrxDa6ZO+i8sNoFbH5w9SAuNqGbtE10hVZadtu0BWbfy6tVs42G22DtEoIpWRRmywuudIKZVEcoELUSCUmyE1QaHEozbISxuYkS2qI4zz3jzPjzBnPzJkzv398P1KUc2bmzHnPnON55n2f931HVBVEREROlqRdACIiyi4GCSIicsUgQURErhgkiIjIFYMEERG5YpCgwERkWESOG/8GjWVHRaTHtt12Eek3Hqv52LJ+r4gcbfJeVREZDljOfhHZaynLXmPZ9iD78/M+tuWByx6yPEP2Y93q9nGUPa3jQcEwSFAgxsmkoqq9qtoLYNRYtRfAA7bN71ZVc/0kgPts66vxlRRQ1VFVfcR4eouqPmIs2xnX+5hBM4wo9kEUFoMEBVUBUDOfqGrN+H8EwMLJTUSq1u2Mx6ss63sAjMddWItKQu9jD5Rp7YMoFAYJCsQIBv1GU9OQbfW4pclpEPXahdUvLesfAHDI6T3MZihrU5SIVETkeWP5kLGs31i2V0R6HJ5Xjaax7QD6LMuHjdeb+zObzKrGNs8bQc5apqOWcrxnPO4x9m++z7BxbPaKiBmUeoznxy3LrPu1l7lhHy6fu2o8b9ivsex5GDU2+2vtn8++vRtju6r9mNn2d7tLmRqOMeWIqvIf/wX+B6AfwFEAey3LBgEMG4+P2rY/jnrzkrn+edSv7u3bDQHYbnm+8Brrvoz/9wLosSy3P6/aywOgB8AwgO0AhsyyWLb/BPXmNPvn3W585iHjfSrGsh6n97Hsz3zfYQD9DvttKLPTsXP43Iv2a/s82+3vZTn+n1jK7rV91dj382b57MfMtr9mZTK3H3b6bPyXvX9LQRSC1nMNoyJy3LJsxLii/nvUcxD210yaV/gAfumy614sroEAqCefAXTgSi5jGMCwcdX6iMNzL2sBrBKRXgAVy1X+v6rRhGYzYuyzYpTvftRzLjvttQ4b8zhMw7nJq6HMqrrouDl8bqf9rjX21ey1/6qqNRFx3N5mO+pNgubxaDhmAK627G+VS5ns21NOsLmJAjFO8mZTgtPJcRxXrkCdmOtHXNYfR/0q1P6+2wFMqupjAGaAetBR1fuM/T1if97koxwHcEjryey7LYHBKUDAOHlXAdRUdRz1YOa4bSualdnpc7t4D1eOW4fHa2tu2zvYB+AuXAnaDccMwHl4HwOn7SknWJOgMJ43rhyBxe3Ze1FvvnE7SR8y1k86tdGr6j650jW2BuAxY9UogKdE5G5zW+MkaD5/zP7c6wMY7/O8iDyC+onfs13eMIP6yRWoJ+Edu+8aZW8WpMxtHcts2ceiz+1iH4BjxnYVo2xer3Xa3q5m1BKeF5Eh+zFDi8e42faULaLKWWCJiMgZm5uIiMgVgwQREblikCAiIlcMEkRE5IpBgoiIXOW+C+yqSkW/0tmZdjGIiHLjV++887GqXudn29wHia90duL/HTiQdjGIiHLjpltu+a3fbdncRERErhgkiIjIFYMEERG5yn1OgojI6jKAi21tuCyCMk86JACWqGL5/Hyo2gCDBBEVysW2Nqz68pdR+dKXICJpFyc1qorauXOY+egjXDU/H3g/bG4iokK5LFL6AAEAYhyHyyGPA4MEERWKApkIEI/v2IF7tmzB//zLv8Su3btx5swZPL5jRyzv9ei2bY7LJYImNwYJIqKIPbptG6pr1uDFf/s3/OS55/DwX/1VLO9Tq9Xw6LZt+Nnhw7HsH2BOgohK7uQrr+Dnzz6L8x9/jKuvvRZfe+ghfPWOOwLvr1ar4eVXX8WTe/YsLKtUKvjkk0+iKG6DSqWCJ/fswfhbb0W+bxODBBGV1slXXsGRPXtw6fPPAQDnp6ZwxDi5Bw0U42+9hTtvv911fa1Ww/9+9FGcO3cO9w4M4OFvfQsvv/IK9j/9NFZdcw0efvhhfPLJJw3PN27YEKgsUWCQIKLS+vmzzy4ECNOlzz/Hz599NlRtwkulUsFPnnsOAPCnf/ZnePhb38ILL7yAv/3udxeCwaPbtjU8TxODBBGV1vmPP25puR89Gzfi0b/+a89tdu3ejZmZGZx5/30AwHe+/W08vmMHzp07hyd/+MNFz9esWbPw2sd37MC5Wg1rqlV852/+JnA5/WLimohK6+prr21puR+VSgV33n47nv7RjxaW1Wq1hce7du/GmhtvxPefeALXXHMNAGDNmjX4yXPP4Tvf/jb2/+hHi55bff+JJ/Dknj2JBAggxSAhIj0e64aN/4eSKxERlc3XHnoIS1esaFi2dMUKfO2hh0Lt98k9ezB55gzu2bIF92zZgn/YtWth3Z133IEf7NrV0G111+7duGfLFuz6x3/E4Ne/vui5l0e3bcOZ99+v/3/mTKhyOxHV5Aeui0g/gGFV7XVZ/wmAGQCPqOqo1766169XThVORKYLS5di3R//se/to+7dlDXv/vrXWHnpUsOym2655biq9vl5fSo5CVUdFZEZj03uaxYciIii8NU77ihUUIhaVhPXPcaIyR5V3Zl2YYiIyiqTiWtV3WnUJDqMpikiIkpB5oKEiAyKyKDxdBpA1WGbIRF5U0TenIlhFCMREdVlJkiISMV4OAnAzEesBfCmfVtV3aeqfarat8roQkZERNFLJUgYNYU+S40BAI4BgKqOA7jfWPee8ZyIKFfingXWbebXqKXVu2kEwIhtWa/l8b7EC0VEFJFHt23Dxo0b8f0nngBQH0wX1QR/tVoNf7djB352+HDDJIJxyWrvJiKiRLx86hR+/NprmDp/HtddfTW+edttuHP9+sD7i3sW2CRmfrVikCCi0nr51Cn800sv4XNjsNlH58/jn156CQACBwrOAktUcmNnz+Lw6dOYnp1FR3s7Brq6sGn16rSLRQH8+LXXFgKE6fNLl/Dj114LVZvwkrdZYDPTu4koD8bOnsWBiQlMz84CAKZnZ3FgYgJjZ8+mXDIKYur8+ZaW+9GzcSNefvVVz2127d6Nx3fsaJgF9h9+8APcs2ULzpw5s+h5mhgkiFpw+PRpXJyfb1h2cX4eh0+fTqlEFMZ1V1/d0nI/4p4FNmkMEkQtMGsQfpdTtn3zttuwYmljq/uKpUvxzdtuC7XfuGeBjXvmV6tUZoGNEmeBpSRtP3bMMSB0tLdj5113pVAismt1FtioezdlTS5ngSXKq4GuLhyYmGhoclre1oaBrq4US0Vh3Ll+faGCQtQYJIhaYPZiYu8mKgsGCUpEkbqNblq92rXsRfqcRACDBCXA7DZqNtFMz87imbffxsGTJ/Hp3FxhTqZOn/PAxAQAxPrZGJgaCQBVhXFPmlJTVYQ9CuzdRLFz6jY6r4pP5+YAFGesQRrdYzluY7ElqqidO4e8d8oJS43jsCTkcWBNgmLnp3uoeTLN8xVwGt1jvQJTno9lGMvn5zHz0Uf4eGoKZQ4TgnrAXG77fbSKQYJi19He7utEmfexBm6fs6O9Pbb35LiNxZYAuCrkiZGuYHMTxW6gqwvL29qabhfnyTQJTp8z7u6xbscs78eSsoM1CYqdvdvoymXL8NmlS5i3tJUWYaxBGt1j/YzbYGKbwmCQoETYu40W9cTl1T02rvcD3ANTWj2uqDgYJCgVSZ9Mi8zrWDKxTWExSBAVgFvNjIltCotBgijnvJqU0uhxRcWSWu8mEenxWDcoIv0isj3JMhHlkVeTUho9rqhYUgkSItIP4CmXdT0AoKqjAGpewYSIvMdKbFq9Glu7uxdqDh3t7dja3c18BPmWSnOTqo6KyIzL6gcAHDUeTwLoBzCeSMGIcqhZkxI7CVAYWRxMVwFgDSAdaRWEKA/YpERxYuKaKOd4jwuKUxaDRA3AKuNxBcB0imUhygU2KVFcMhMkRKSiqjUAhwCY916tAhh12HYIwBAAdF5/fWJlJCqToo6Kp9ak1btpEECf8b/pGACo6rixTT+AmvncSlX3qWqfqvatuuaaRMpMVCa8TwWZ0urdNAJgxLas1/J4X+KFIqIFnM6DTFns3UREKeN0HmRikCCiRXifCjIxSBDRIhx7QabM9G4iouzg2AsyMUgQkSOOvSCAzU1EROSBQYKIiFwxSBARkSsGCSIicsUgQURErhgkiIjIFYMEERG54jgJohybPH4cJ44cwYVaDSsrFWzYvBnV3t7mLyTyiUGCKKcmjx/HGyMjmJ+bAwBcqNXwxkh9cmUGCooKm5uIcurEkSMLAcI0PzeHE0eOpFQiKiLWJIgilGTzz4VaraXlREEwSBBFJOnmn5WVimNAWFmpBN4ncxxkx+Ymoogk3fyzYfNmtC1b1rCsbdkybNi8OdD+zCBnBh4zyE0ePx66rJRfDBJEEUm6+afa24tbBwcXag4rKxXcOjgY+MqfOQ5ywuYmoojE0fzjJMomIeu+3DDHUW4MEkQR2bB5c0NOAgje/OMWCKLMe9j35SbqIJdXY2fPNtyEqfu66zAxNVX4mzIxSBSA/cebpx9rkRKlZrnDfh6vQODVJNTq+zjty0nnzTe3tN8iGjt7FgcmJnBxfh4AMD07i1c/+GBh/fTsLA5MTABAbv72/EolSIjIIIAagB5V3emwflhVHxORIVXdl3wJ88PpxxvFjzWJwFPEwWDV3t7QZfcKBFHmPfy+5sN33ml530Vz+PTphb8xNxfn53H49OnCBYnEE9ci0gMAqjoKoGY+txkSkfcATCZauBxy+vGaP9agzMAzPTsL4ErgGTt7NlRZ7ZgodeYVCNyafoI0Cfl9DXMSWPhbiGq7PEmjd9MDqNcigHoQ6HfY5j5VXWsEEvLg9qMM82ONI/A44WAwZ16BIMpur077aqU8ZdLR3h7pdnmSRpCoAJixPO9w2KZHRPpFZHtCZcottx9lmB9rHIHHSZRXxUXiFQii7PZq7mu5x28lzLiLIhno6sLytjbPbZa3tWGgqyuhEiUnk4lrM08hIneLSL+9RiEiQwCGAKDz+utTKGF2DHR1NeQkgPA/1o72dseAEPVVUpS9gYqkWQI8iryH9b1OHDmCiw7ft4iEGndRJGaegb2bklEDsMp4XAEwbV1pJLWhqiPGuqp9B0Yyex8AdK9fr3EWNuucfrxhf6xxBB4nUfUGKqIoA0Ezbs17qsrvwmLT6tWFDALNpBEkDgHoMx5XAYwCgIhUVLWGep7CTFivBbA38RLmTNQ/3jgCj5skT4a0mNeUG0Ga/YrUpZnqEg8SqjouIn0i0g+gpqrjxqpjAHqN9UMiMgPgPct6R5/NzeHU1FTDsvXXXRdL2cukrFdNZePVk6zVZr8idmmmlHISTmMfVLXXa72bZVddhRvWrVt4/p/vvrsoaADFChx5HjwXpyBXsV6vKcNVsVdPsigG51kH+pXheBZRJhPXYVgDhsktcJjyFEDiGjyXd0GuYr1eA6AUV8VRzjfl1aWZtYz8KlyQcOIUOKxOvfuu4/IsBg+vMQxlDhJBpqtoNpgvqukvktTq1XqUPcy8Ao7bsf7FwYMAGCiyrBRBoplWah9pB46kxjDkTZCBeVG+JguCXK1H2cPMK+C8/tOfOr5GVVmjyDgGCRdZCxxmHsJNEUd6tiJIs0mz1yQx7XeUgk7+F1UPM6+A4zXvVJI1NObzWscg0YJmgSOuYGHPQ9jFOdIzL8nGIM0mzV6Tt4F+WZjmxC3gOB1rqyTKyHxeMAwSIZmBI87kuNcMlF9ctgwKYP+JEzh8+nSkV0Z5Sja22mxiBr/5uTmICFTV8TV5CJCmpG56FIR53H5x8CBUF49/TaKMzOcFwyAREa/keNhmKq98w8XLl2O7Mory3gVJ8NtsYg9+qtowN1Kr+8uKrE9zYh7LtMrIfF4wDBIJCJvfcJtLaYlIrFdGWWi+iEPegp9fbrUpAHjhe9/LRI0ozalYkpqTrGgYJFLSSuBwm0vJrQkqqiujLDdfhFHU4Acsrv1ksckwrRpaUnOSFQ2DRIa4Nlm9+y76b7wRr/3ud/ivixfxB8uX47Y//MOF53ZRXRllvfkiqKwFvzg7BxS11hREknOSFQmDRA7csG4dbli3Dnfaln9pchLPj43h0uXLC8uWLlmC/97ZGcn7FnWW1iwFv7iv9ItcawqCc5K1jkEix3qr9VnUj5w4gdqFC6isXInNGzag89IlnJqaiqRLbt6St35kKfjFfaWftVpTlnDMhD8MEjnXW60uBAurrI4Yz4o0gp9Ts1LcV/pZqjVlCcdM+McgUVBu+Y08zVNVJG7NSiu+8AV8/vvfL9o+qiv9LNWasoRjJvxjkCiZKGfJzcto7Cxwa1ZqW7oUbcuWxXqlX8Qmw7A4ZsI/BgkKNBDwqg8+yFzXyixzaz66ODuL//Hggwy2CeOYCf8YJMiTW83jly++yK6VLfBKIBftSj8PNUyOmfBvSdoFoPy5Yd06XDx/3nHdhVoNp6amFv5R3YbNm9G2bFnDsiImkM3cixkQzRqm172007Bp9Wps7e5eqDl0tLdja3c38xEOWJOgQNo7OjA7Pe243Fr7MBPlZU+MFzGB7FRjyNPgPY6Z8IdBggLpGhjAxIEDmLeM+G5bvhxdAwMN292wbl2hbh8bRpGaldx6a6U5FTjFw1eQEJE1AL4OYERV3xeRe1X1haBvKiKDAGoAelR1Z6vrKTpnx8Zw+vBhzE5Po72jA10DA1i9aVPT15nb+HltnDPkUjrcagzmtOt2HLyXX35rEoMARgHcJyJHAdwNIFCQEJEeAFDVURGpikiPqo77XU/ROTs21lAbmJ2exsSBAwDgO1D42c5L1u4ASP641QzMadc5eK84/AaJUVV9C8BbInIXgMVDfP17AMBR4/EkgH4A4y2sp4icPny4obkIAOYvXsTpw4dDn/zD4EDA7PPqrWXmJoqSeyk7v0GiZjYxqeoxEQnznhUAM5bnHS2up4g4JZ69lqctyoGAFI7XdB95z704zekElHf22KZBQkSuVtUzAM6Yy1T1WKylakJEhgAMAcD1Ec14mrbjk5OLJupzmpMpSm49lJatXBnr+0aJ+Y50FLG31tjZszh48iQ+tQS+6dlZPPP22wCAeSPXUrZ5nvzUJP6PiBxS1RMishGAquqJEO9ZA7DKeFwBYD9LNVsPVd0HYB8ArO/uXpwly5njk5MYeeMNzBkDe2oXLmDkjTcAINZA0TUwgLefeQZqm8Pm0mef4ezYWKpNTlHwqnkwUISX9xqDlX3CP6t5h0R8meZ58jOY7k0AVaNG8RaunMCDOoQrOY0q6glxiEjFa32RHTlxYiFAmObm53HkRJhY3NzqTZuw1GEaAp2fx+nDh2N977SY9+awDvjj4D9ymvCvmbLM8+SnJlFF/ep+p9EV9iiAl4O+oaqOi0ifiPQDqFl6Lh0D0OuxvrBqFy60tDxKc59+6rg8q3mJqLBXVXTyMA1HM0FO+GWZ58lPkJhU1Z8BeAoAROTesG9qNBfZl/V6rS+yysqVjgGhkkBuwGvkdNkwMd66LN5DOwi3Cf8AQADYG5zKNM9T0yChqj8TkRuNQXQbAaxNoFylsnnDhoacBAAsa2vD5g0bAu/Tmgg3Bzg5JcT9jpwuKybGveVpGg4vThP+AcCKJUtwSXVRXuJPv/KVUuQjAJ9dYFX1feP/twC8FWeBysjtNqRBk9b2RLg5AtYpId7KyGlqxCar4txD2zzh27u5ms/tJkqUv+LcTRnhdhvSIJwS4SYzIW59ryhGTlNd2QYCFuke2k4T/u136TxSlqQ1wCBRSM0S3kkkxKlR0O64WU8KF/0e2rw5EYNEIbklwq3rKX3NZsidPnkSv3vppUwnhYs0qM5ppDVvTsQgUUhOiXBT2IQ4RcsrMT7xz/+ci6RwEQbV2QfTmaOqt3Z3Y2t3d2mn5AAYJArJngj36t1E2dXs7n92ec9vpMlpMJ05qnrnXXeVKijYMUgUVJSJcEqH37v/AeXrVRU1t0R0mRLUbhgkiDKqlTEsHAgYDhPU7hgkEpTGTK+UX2HHsHAgoH9MULtjkEhIWjO9Zl3Q26eWRVxjWJrVPMoWLNwG05U5F2FikEiI10yvZQ0SYW+fStEyA0dZaxlOg+mIQSIxac70mlVZvX1q2XG6EbJikEhImjO9ZlWebp9a9mYxJsbLi0EiIXHM9Jp3eZmmnM1izsImxrM+5QjVMUgkJOqZXovAqYsnAFzX3Z1SiZyxWayRn1pVs4kOp0+exG///d+hly4ByOaUI1THIJEgDnBrtHrTJsz85jf44NVXG5b/xy9+gVU33RTLCThIs1GemsXiFrZWZQaPU/v3LwQI0/zcHH754ov47I/+KPWmKqd5nMqa1GaQoFRNTUwsWhbmKt0rCAQ9weWlWSwJUdWq3ALsxfPn6/cgN2ocaQQLt3mcAJQyUCxJuwBUblFepZtBwHytGQTOjo0B8D7BeekaGEDb8uUNy8p6976ovi+3AGsuv2HdunqwmJpy/Bcnr3mcyog1CUpVlFfpza5yg57gePe+K8J8X9Za3rIvfhHS1ga1nIydAm8avao4j1MjBgmKhd+2/yjvsd0sCIQ5wfHufXVBvy97U9/cp59C2tqwbOVKzF240FLgjXu6Ec7j1CiVICEigwBqAHpUdafD+mFVfUxEhlR1X/IlpDCc2v5P7N+Pmd/8Bn/yjW80bBvlVXqzIBBlQCqroN+XUy1P5+ex9Kqr8Oc//GFk5YtiICDncWqUeJAQkR4AUNVREamKSI+qjts2GzICySNJl4/CczohAMAHr77q2Gspqqv0ZkGAzUbRCPJ9pdlDrNXA0co8TmXoBZVGTeIBAEeNx5MA+gHYg8R9qjqaaKkoMl5/+HGOLfATBNhslBxrk6MsWQK9fHnRNlH1EGt1huVm4zj8zONUll5QaQSJCoAZy3OnX0mPiADuzVFDAIYA4PrOzjjKSCG4NfsA8V85Mghkg73J0SlARNXUF+UMy83uOw5caaby6gXFIBEzMzCIyN0i0m+vVRh5in0AsL67W1MoInnoGhjAif37HdeVcWxBUXl1TnBrcjRrFFE29UU9w7LfxHhZekHFEiSMK327SeNkXwOwylhWAdBwaWnkIqCqI8Y6DlEOIY0bHbmNpGaSuDiaDUx0qzHq5cv4i6efjrQsSc6wbA0glV/9yvE9/mD5cpyamkp91HhUYgkSTXokHQLQZzyuAhgFABGpqGoN9TzFpLF+LYC9cZSxDNK80dGffOMbWHXTTZlIEpd9Btc4NBuTkuQo9bRmWHabtPMvbrkFN1SrC/kNu7wFj8Sbm1R1XET6RKQfQM3Ss+kYgF5j/ZCIzAB4z6HnE/mU9o2OspAf4Ayu8WiWc/LqaRZ10E5rhuVmk3YWZXr1VHISTjUNVe31Wk+t442OOINrXJrVFNx6mgGIPGinOcNyq5N25vG+45lMXFNzfnINXtXwNHIVYQS9+uQMrvHwMzDRqSZ5bPv2WIJ2EWZYzup9xxkkcshvrsGtGn5zZ2dquYogwjQZcQbXeAQdmMig3Zos3HecQSKH/OYa3KrhaecqWtWsycirlsGpOOITJOfEoB1MmvcdZ5DIoVZyDU7V8J++/rrr67/7L/+SueYnr6vPZrUMTsWRLQza0UkqMc4gkUNhu/y5vd6UteYnr6tPP4npLPSyojoG7XgFSYw3wyCRQ2G7/Dm93i5LzU9eV59uI7vZxp1dDNrp8AogXnhnuhzqrVYxeOutCzWHysqVGLz1Vt8ndPvr3WSlq+zqTZvQvXXrQrt1e0cHurduXRi05YRt3ETRYE0ip8J2+bO+/nsvvJDKiNVWWK8+zUT1if37fd/hjJKVty7W5I5BglIbsRpElHc4C1sOv+3qZZsWJM3pYCh6DBKU6ojVViV1hzMvZ8fG8PYzzyzUXmanp/H2M88AWDxuo4zTguStizV5Y5AgAPkZsZqFwVgnDx5saN4C6oHq5MGDi078ZZwWhNPBFAsT15QrWUhUz336qe/lWQhqSTo+OQnjhmGLZCnHRf6xJlEQZUkU5m0wVplGGJu5CNXF9wHLao6LmmOQKIAkE4VpB6MsDMYyk+ROy+3yFtTCcMpFAICItNRFm7KFQaIAkkoUZqXXStqDsb764IMNiWsAkLY2fPXBBxdt6yeoFaX3k1vOQVUZIHKMQaIAkkoUstdKXau1Ga+gVqTeT2ndIY7ixSBRAEn9cbLXyhVR1WaK1PspT+NtyD/2biqAzRs2YFlbW8OyOP443YIOrxSDK1Lvp7DTxVA2sSZRAEkNhnObGPDmzs5I36dMitb7KS/jbci/1IKEiPSo6rjLukEANQA9qroz2ZLlUxJ/nL3VKt7/6COM/frXDcvfnJwEALzz4YeF74IbtTL1fqJ8SiVIiEg/gGEAvQ7regBAVUdFpOoVTCh573z44aJlc/PzDYGjKHP1JNHdNwtdeom8pBIkjAAw47L6AQBHjceTAPoBMEhkhN8kdd57PSXZ3TftLr1EXrKYuK4AsAaQfDbOFlQrSeo893ry6u5LVCa5TFyLyBCAIQC4nknT2FmbXdqXL0fbkiWYv3y56evy3OuJ3X2J6mIJEsZJ3G5SVUd9vLwGYJXxuAJgUdcPVd0HYB8ArO/uXjxRDEXG3uwye/EilojgCytW4Peff47KypW4ubMTb05OFqp/fFkGhqU9zQplXyxBwjiJt0REKqpaA3AIQJ+xuArAT2ChmDg1u1xWxfKlS/F/779/YdmNX/5yoU42ZRgYlpVpVijb0urdNAigT0QGVXXEWHwMQK+qjotIn9EDqsaeTeny2+xStP7xeboRU1CcZoX8SKt30wiAEduyXsvjlmsiFI+yNLs4KVrgs2t2AcCmKAJymrim5KTd7MITVXy8LgDYFEUmBgnylGazS1InqrIGIq8LADZFkYlBgppKq9kliRNVma+YvS4Afvr6646vYRfg8mGQoMxKYqxC2a+Y3S4AypyLokYMEpRZSZyoyjJortUmtbRzUZQdWZyWgwh44FXYAAAFe0lEQVRAMvfJKMM9MswmNTPwmU1qx43Ze53w3hBkYk2CMiuJpHkZrpiDNqkVvQsw+cMgQZkW94mqDIPmytKkRvFgkKDSK/oVM5PQFAaDBFFBWZPVds2a1Mo6doQWY5AgKiD7+A+rZif9Mo8docXYu4mogJyS1UA9QPzdvfd6nux5wyWyYpAgKqAwyWomusmKzU1EEcpKW36YZDUT3WTFIEFkE/REn6W2/DDjP8owdoT8Y5Agsghzos/SPFBhxn+UYewI+ccgQWQR5kSftbb8MOM/ij52hPxj4prIIsyJvgzzQFH5MEgQWYQ50ScxISFR0hgkiCzCnOg5cyoVUWo5CRHpUdVxl3XDqvqYiAyp6r6ky0blFTZpy7Z8KppUgoSI9AMYBtDrssmQiAwCeCS5UhHV8URPdEUqQUJVR0VkxmOT+1R1NLECERGRo6x2ge0REQDoUdWdaReGiKisMpm4VtWdRk2iw2iaIiKiFMRSkxCRIYfFk36akIxcBFR1BMA0gEWNw8b+hwDg+s7OcIUlIiJXsQSJID2SRKSiqjUAk8Y/AFgLYK/L/vcBwPrubg1RVCIi8pBKc5NRW+gzaw2GYwBgdIu931j3nls3WSIiil9avZtGAIzYlvVaHnNsBBFRBmQycU1ERNnAIEFERK4YJIiIyBWDBBERuWKQICIiVwwSRETkikGCiIhcMUgQEZErBgkiInLFIEFERK4YJIiIyBWDBBERuWKQICIiVwwSRETkikGCiIhcMUgQEZErBgkiInIlqvm+RbSITAH4bcrFuBbAxymXIQiWO1l5LHceywyw3M38N1W9zs+GuQ8SWSAib6pqX9rlaBXLnaw8ljuPZQZY7iixuYmIiFwxSBARkSsGiWjsS7sAAbHcycpjufNYZoDljgxzEkSUeyLSo6rjLusGAdQA9KjqzmRL5q5JmYdV9TERGVLVVAMHaxIREJEej3XDxv9DyZXInyblHhSRfhHZnmSZiqDZscvqsfVR7kz+lkWkH8BTLut6AEBVRwHUvH7zSfIqs2FIRN4DMJlQkVwxSISUpy/bKo9/WED2T2TNjl1Wj63PcmXyt2yUecZl9QOo1yKAern7EylUE03KDAD3qepaY7tUMUiElKcv2yqPf1g5OZE1O3aZPLbwV65M/pabqKDxd96RVkFa1JOV2iaDRPwy82W3IKt/WHk4kTU7dlk9tn7Klcffci6p6k7jN9xh1PpTwyARsyx92QXAE1mKcvpbrgFYZTyuAJhOsSy+GE2qg8bTaQDVNMuzNM03zwOXtu1JP1eq5hetqiNI+MsOU27k8A/LZPZeEZG7RaQ/hRpFs2OX1WPrWa40f8tBiEhFVWsADgEwRzBXAWS2qcxS5klcaS5dC2BveqVikGgqSPezLHzZIcud2h9Wk+CWhxOZ47HLwrFtolm5M3XisjK+9z4RGTS+ewA4BqBXVcdFpM+o+dTcupwmzUeZh0RkBsB7aZeZ4yRCMr7spwD8L/PLFpHjqtprPB5CvYmkmrE+2n7KPYl6uTMxwMdIVPep6j6jOWnU+IOqqKqZyJ40Hu8FsDeNPzCnY5f1Ywv4LnfmfssULwYJyhWeyIiSxSBBRESu2LuJiIhcMUgQEZErBgkiInLFIEFERK4YJIiIyBUH0xFFxBh7UkW9i+4tAP7eGIhGlFusSRBFQESqxqBEMygcMgOEiPTkaK4jogYMEkQRUFVzyopeGCPBLasfQH0aEaLcYZAgioDl3hZVyxQhpkNplIkoCsxJEEWjX0SqAI4aTUteN6Iiyg1Oy0EUM2M+qbVgIptyiEGCiIhcMSdBRESuGCSIiMgVgwQREblikCAiIlcMEkRE5IpBgoiIXDFIEBGRKwYJIiJy9f8BpKcfy+qNJC4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cfadc6c5c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_contour_2d_classification(circle_X, circle_y, lambda x: circle_svm_standard(x) > 0,\n",
    "                               delta, \"SVM classifier with standard kernel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is unsatisfying. However, as we know that all vectors in the subset of the data set labelled -1 have a length smaller than 1 and all vectors labelled 1 have a length greater than 1, a feature map introducing a feature that is related to the length of the vectors should be able to linearly separate the subsets based on this new feature. Therefore, we introduce the square of the length as a new feature using a feature map and train another support vector classifier on the mapped data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def feature_map(X):\n",
    "    \"\"\"Maps 2 dimensional training set into 3 dimensions\n",
    "    :param X: training set\n",
    "    \n",
    "    :returns: ndarray. mapped training set.\n",
    "    \"\"\"\n",
    "    return np.concatenate((X, (X[:, 0]**2 + X[:, 1]**2).reshape(len(X), 1)), axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "circle_mapped_X = feature_map(circle_X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "circle_svm_mapped = SMO(circle_mapped_X, circle_y, circle_C, max_iter)\n",
    "circle_svm_mapped_classifier = create_classifier(circle_svm_mapped, classes, borders)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAERCAYAAACO6FuTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X1wFOedJ/Dvg9AL1pYZBOQc4vOiIect46N20OAqwXqXNSvVropyVSwPeCFOYCERVYl8twmOzJYgVXeGCmArLq/5R8SmYJ1AACEnlfNSG0CESni5ChL6w5haXB6xzq6TNTYMrgMkBHruj+keelrdPTM9/T7fTxWFprun55memf718/ZrIaUEERGRkSl+F4CIiIKLQYKIiEwxSBARkSkGCSIiMsUgQUREphgkbBJC7BBCDCr/UsqyY0KIJt12XUKIFuVvqf6tWd8rhDhW4LXiQogdNsvZIoTo1ZSlV1nWZWd/xbyObrntspdZng79sS51ezfK7tfxMCiH4ecVRVbfBWXdYfU3TJMxSNigfOFiUsqklDIJ4LiyqhfAc7rNW6WU6vo0gBW69XH3SgpIKY9LKTcoD5+QUm5Qlu1063Wc+MHxR+s87THVfS8qWauUcoWUsq+YjSvxe8kgYU8MQEZ9IKXMKP/3Ach9iYQQce12yt8NmvVNAIbcLqxGzKPX0QdKv/ZB+XhMJyv1N1Fxx5BBwgYlGLQoTU0dutVDmianFLK1C63fatY/B+Cg0WuozVDapighREypGh9TX1dpNjisbN9k8DiuNI11AVikWb5Deb66P7XJLK5sc1gJctoyHdOU40Pl7yZl/+rr7FCOTa8QQv0BNimPBzXLtPvVlzlvHybvO648ztuvsuwwlBqb/rn696ff3oyyXVx/zHT7+0uTMuUd4wKvo26rNhHGlX31CiE+1H4mFuVIGBwv/TFVPy+z43hYs9ywadKkrJP2pdlev15t6jls9FmZvX+zY2LwXbb8bMX938Sk34JJWYyOobYpV/85xHX7zPueF/ouBIaUkv9s/gPQAuAYgF7NshSAHcrfx3TbDyLbvKSuP4zslYx+uw4AXZrHuedo96X83wugSbNc/ziuLw+AJgA7AHQB6FDLotn+OrLNafr326W85w7ldWLKsiaj19HsT33dHQBaDPabV2ajY2fwviftV/d+uvSvpTn+1zVlt9o+ruz7sFo+/THT7a9QmdTtd5i8ty4AKc33qEPZ/rBm2Y5C5TA6Xiafyw6TMrdA+f6pr11CWU0/a2X9oOb5hr8Tg8857/2bLNMfD8vPVn9M9M8v4Rj2ao+HxffrMAy+52H4NxVkm8z2NRwXQgxqlvUpV2g/QLYPQv+ctFCu8AH81mTXSUyugQDIXf3MxP2+jB0AdihXbRsMHluZB6BBCJEEENNc+R2SShOaTp+yz5hSvpXItunuFLpah456HD6DcfU+r8xSyknHzeB9G+13nrKvQs89JKXMCCEMt9fpQrZJUD0eeccMwIOa/TWYlEm/vZknAOxW/h5SynYcwDUg991SP1PTcpi8Zyv6MqeVfcTU1y6hrIU+6/PK/xnNthkhREw5hkblNnr/+mVp5B+PxwH8rwLvW8vot9BhUJZiaL9f2jJtA9Bt9T0PIjY32aCc5NVqudEXSP3RHDbZhbrerLNsENmrOf3rdgFISylfwv0fSVpKuULZ3wb94wJvZRDAQZntzG7VBAajAAHlSx0HkJFSDiEbzAy3LUWhMhu9bxMf4v5xm2nx3IzZ9gZ2A/gr3A/aeccMwOewPgZG25v5raY8Lbh/EdGgvJcW3D+xmpajhONlpRXAj2AeRM3KaptFuY3ev36Z/nhcROHPVkv//A6TspiZp/lb/T7of1/DJfw2A4M1CfsOK1eOwOQ2z15kq6xmX4SDyvq0vt0WAKSUu8X9obEZAC8pq44D+JEQolXdVvlhqY9f0j+2egPK6xxWrsQyyhe4kGvInlyB7A/VcPiuUvaifghmZdbsY9L7NrEbwAllu5hSNqvnGm2vl1GuCg8LITr0xwwlHmOr7ZUa2WEhxD8ge4JaoVyExJW29TiyAavQfk3fcwmfizroYocQ4iX9Va9FWcthVm79+2/QL9N8Rurx+CYKf7ba96M/nj8wKUvuGCq/30VKv8Sk926wz9+iyN9mkAilvYyIAkg58b5kccHhxmseVl4zrVypr/Dy9XVlmfT+/TgmlYw1CSLS+wGytVL1cUVMuiNjrEkQEZEpdlwTEZEpBgkiIjLFIEFERKZC33Eda2iQc770Jb+LQUQUGpfee+9TKeXsYrYNfZCY86Uv4Z9+9jO/i0FEFBpPfPnL/1bstmxuIiIiUwwSRERkikGCiIhMhb5PgogCYGICVXfuQExMAJygGwxCQE6Zgns1NcAU+/UBBgkiKlvVnTv4QkMDpsdi0KTzIB9JKXEjk8En167hXl2d7f2wuYmIyiYmJhggAkYIkf1MJibK2g+DBBGVT8pABIgt3d14evlyPL96NV7r6cHIyAi2dHe78lovdHa6sl8nCSHKbv5jkCCiSHihsxON8Th+8e67+PH+/fi79etdeZ1MJoMXOjvxzpEjruw/aNgnQUSee39gAL/euxefX72KB2fPxp+vXYv5y5bZ3l8mk8GvBgbwxq5duWWxWAzXr193orh5YrEY3ti1CxeGhhzfdxAxSBCRp94fGMC/vP467o6NAQA+/+QT/MvrrwOA7UBxYWgIf2nx3Ewmg85vfQs3btzAM+3tWLd+PU4ODOCtN9/EjIYGrF+/HtevX897nFi40FZZooZBgog89eu9e3MBQnV3bAy/3ru3rNqElVgshh/v3w8AeHLJEqxbvx79/f148XvfywWDFzo78x5TFoMEEXnq86tXS1pejIVNTfgfBTqSX+vpwbVr13BlZAQA8N2NG7Gluxs3btzAP+7aNelxY2Nj7rlburuRuXED8cZGfGfjRtvlDCN2XBORpx6cbZx81Gx5MWKxGP5y2TLseeut3LJMJpP7+7WeHsxtbMTL27ZhxowZAIDGxkb8eP9+fHfjRux5881Jj7Ve3rYNb+zaVXEBAvAxSAghmizW7VD+7/CuRETkhT9fuxZTa2vzlk2trcWfr11b1n7f2LULI+k0nl6+HE8vX46eV17JrXtq2TL0vPpq3rDV13p68PTy5fhhTw+eTaUmPbbyQmcnroyM4IXOTowoNZOo8uUe10KIFgA7pJRJk/XXAVwDsEFKedxqX/MXLJBMFU7kr6k3b+K/Pfpo0ds7PbqJzH1w+TLu1tfnLXviy18elFIuKub5vvRJSCmPCyGuWWyyolBwIKLwmr9sGYNCSAS147pJmb3ZJKXc6XdhiIgqVSA7rqWUO5WaxEylaYqIiHwQuCAhhEgJIdReo88AxA226RBCnBdCnL9+zarVioiIyhGYICGEiCl/pgGo/RHzAJzXbyul3C2lXCSlXDSjocGrIhIRVRxfgoRSU1ikqTEAwAkAkFIOAViprPtQeUxEVJDbWWDDkPnVaX6NbuoD0KdbltT8vdvzQhFRqL3Q2YmFCxfi5W3bAGQn0zmV4C+TyWDL5s1458iRvCSClSCoo5uIKMJOXryIfadO4dPPP8esBx/EmqVL8dTjj9ven9tZYCst86sWgwQReerkxYt44+hRjN29CwC4+vnneOPoUQCwHSiYBdY9DBJEJRpMp3F0eBiZmzcRq69HWyKBZHzSIDwyse/UqVyAUI3dvYt9p06VVZuwwiyw9gVmdBNRGAym0+g7dw6ZmzcBAJmbN9F37hwG02mfSxYen37+eUnLi7GwqQm/Ghiw3Oa1nh5s6e7OywL76iuv4OnlyzEyMjLpMWUxSBCV4OjwMMbv3ctbNn7vHo4OD/tUovCZ9eCDJS0vhttZYCsZgwRRCdQaRLHLabI1S5eidmp+S3ft1KlYs3RpWft1OwtsJWV+1fIlC6yTmAWWvLS1v98wIMTq67G5vd2HEgVDqVlgnR7dROZCmQWWKKzaEgn0nTuX1+RUXVWFtkTCx1KFz1OPP86gEBIMEkQlUEcxcXQTVQoGCfJElIaNJuNx07JH6X0SAQwS5AF12KjaRJO5eRMHz5zBz8+fx62xscicTI3eZ9+5cwDg6ntjYCI3cXQTuc5o2OiElLg1NgYgOnMN/Bgey3kb5DYGCXJdMcNDozDXwI/hsZy3cd+W7m48uWQJnlyyBO/09/talqeXL8fwhQt5y17r6cHJAhP+gohBglwX0w2/MxP2uQZm77PY928H521knRwYQObGDfzmzBn85swZPOXR/bPNgtG69etxpC8v0TUGBgaKLpffQU6LQYJc15ZIoLqqquB2bp5MvWD0Pt0eHutHYHKSci/7smUyGcSmT889jsViFls7Rx8IVM+0t+Nn77yTezwyMoLpmvLZ3a8fGCTIdcl4HKnm5tyJa1pNDaqm5H/1ojDXQP8+Y/X1SDU3u9qJXExgGkynsbW/Hy++/Ta29vcHpr9ialUVHqitxdQiLiAKeaa9HSdPnsSTS5bkpeYYUWZIA9nmnnf6+zEyMoInlyzBC52dWDB/PkZGRgyXAcDzq1fj6eXLc/tQ9/f86tV4obMTvzp5Ei90dualAFH9aSKRa3L6WX8/1n/jG7l16n7VGsMLnZ25meJbursn7deqHPt/8pNcefRNXE7g6CbyhH7YaFRH5FgNj3Xr9QDzeRt+jbgqZGpVFWqU1Bzq/3d1fSul+s2ZMzg5MIAf9vTgwoULljcHmjt3Lt7YtQvv9Pdjz5tvYt03vjFpWUNDA55NpfBMe3t22Vtv4ally/DOkSN479IlxGIxXBkZMX2dZ1MpHOnrQ2LhQgwMDOA7GzcCyAarZX/1V1i3fj2eX70a169fR7yxMW8/w8PDucevKWlCzMqxZfNmV7PXMkiQL7w+mUaZ1bG06tj26/gLIVBbXT3p8b2JCZSbJuipZcvw1LJleHLJEsvtZjQ0AMjWQNSah37Z9OnT8Xfr1wMAEgsXYkt3N55atgzPPPtsUc1Zz7S34/ubN2Pj976HuY2NueXpkRFcv3YNFy5cwI0bN3Du7Fl869vfNt3P4OCgZTm+u3EjtnR348aNG/jHXbvQqHktJ7C5iSgCzJqUgtixLaXE2Ph4LiDoH9sxfOFCrmnGLPleWrP8+rVrALId3uoJXL8smUzmRiOdHBhAMpm9w3KshL6FP00ksGXzZrRr8notXLgQz6ZSeGPXLvzi3XfRvHix5ainQuVwO3staxJEIWfVpBSrrzdNSOgntWmptroad+7eLbupCQC+9tWv5m5X+vZPfgIgewK9MDSUy+CqunLlCp5fvRpXrlzB//nnf8b169cnLYvFYnh+9Wr0vPoq5s6dix/v328YgJ5evtz0Cn79N76Br331q3lNSWozk1pb+fH+/bk+CfWxdr/f2bjRshyv9fRgQAkiL2/dWvZx1PMtC6wQoklKaXjDWCFECkAGQJOUcqfVfpgFliqdVWZas4SETneol5oFViWEKLuJqVQjIyP4YU9P3onbaFlUlJsF1pfmJiFEC4AfmaxrAgAp5XEAGfUxERmzalLyY8RVKcJ+q4JK4Etzk5TyuBDimsnq5wAcU/5OA2gBYFjjIKLCTUocJJCvUTeSyGwZZQWx4zoGQBtAZvpVEKIw8GMSH1UOdlwTOeD3ly+X/Jwv2mjDN8J7XJCbghgkMgAalL9jAD7zsSxUYeyc7FXzZ88uetv3r151NLCwSYncEpggIYSISSkzAA4CUHvd4wCOG2zbAaADAB6aM8ezMlK0mJ2kSznZ22XnNQoFFqdqJip1VnxddTVGx8cDXTv5o2nT8It3381LoKcOe/3Fu++WvL+TAwPo7+8vqp/i6eXL8fLWrXkznl/r6UFi4ULPEg26yZcgoQxxXSSESEkp1UxWJwAkpZRDQohFygiojNEwWSnlbgC7gewQWM8KTqHlZ0BwilVZzQKI3cAxmE7j0OnTeOT2bezZvh3rNm3CodOnAfibzsNMY2Mj+vv7807KV0wm1RVDnbldDDXjqzZIaNNwFPJOfz+e0Uy2Cxq/Rjf1AejTLUtq/t7teaEoEqIQDOwwen/lBI6jw8N45PZt7N25E3t3Zqcqre3q8jWdh5XpsVhuxjSQnYGdSCQw7MF9NdT0Gy9v2wbAXsbXIAeJII5uIrJFPSHOnz170r9KZHYcfn/5csH+kLrqauzZvj1v2Z7t21GnyblUjrt37+LA/v2QUuLA/v24e/du2ftMJpO5LKhH+vrwbCqVW5fJZHKZVPe89VZeFtWRkZG8LKuv9fRgZGQEW7q7MTIyklv+5JIlhtlegehkfDUSmD4JolLpT3SVGgxKoR4jfS1DX7sYHR/Huk2b8pat27QJo7NmOVKOw4cO4Zvr1+ObSuI6AFi1enVZ+/xKezv2vPkmEgsXYmRkJC+pXiwWy6W7eHLJkrwsqheGhpBMJnPpL76za9ek9Btv7NqFLd3duDA0ZNgMFZWMr0ZYk6BQUa+CjWoNVDyz2oV6XNsSCXw0bRrWdnVhYmICa7u68NG0aY7NvfjbVassH9vR2NiI4eFhjIyM5JLgab3W04Mt3d25vgo1i+qMGTMAZGsbaiZYLTXYNDQ0mNYk1JsMZTKZSRlfB06cwAudnbmMr1Z9HYODg7n1iYULMXDiRF5Zv7txI1595RU8vXy5aSJDpzFIUOAxMLhL3yT1+8uXMefuXfz1vHn4dPp0rHr9dVyfNQsr/+zPHOuP+OmBA5aP7UokEtjS3Y2v6Nr4X+vpwdzGRry8bVsuKGizuQ4MDKDzW9/Cd4vsbDYShYyvRtjcRIGlDQrkHfV4z589G48pzUtOD69dsXIlgGwN4qcHDuQel+vZVApf++pX0djYmHfV/9SyZej89rdz2VL1royMYHoshi3d3bkO6FJFIeOrEd+ywDqFWWCjhf0MwfT+1at5j/VBw24W2CB4fvVqvLxtGxobG0uaHxEW5WaBZU2CAkEbHBgYgkf7maid3k7XLvzy4ve+hx/29OQer9d0phODBAUAm5XCJTdCSvncwh4sEgsXRqrm4DQGCfINg0O4zZ89O1er+K8PPwwpJYQQfheLNKSUQJmfCYMEeYp9DtGifn5Xb93CZ598ggenT4cQAtW1tT6XjKSUuJHJQE4pbxArgwR5grWGaJtZVYX/d+0aPr9+HXcmJgAAVVN5evGVEJBTpuBeTU1Zu+GnSK5jgIi+KQDqlPtoP6Ase/8//gNA+PssKh2DBLmGwcF96cFBDB89ipuZDOpjMSTa2hA3mG3sB22fBcBgEVYMEuQ4Dmf1RnpwEOf6+nBvfBwAcDOTwbm+bHLlIAUKIHrDZisJ03KQo/SpM8g9w0eP5gKE6t74OIaPHvWpROZKyUBLwcKaBDmCtYcsL5t/bpokmzNb7jfWKsKJQYLKxr6HLK+bf+pjMcOAUB+L2d6nF0FO7augcGBzEzmi0gME4H3zT6KtDVW6mwBVVVcj0dZma39qkFMDjxrk0oODZZfVCJuewoFBgmzjjzyf180/8WQSzalUruZQH4uhOZWyfeXvZZDT38uCgovNTWQLm5gmc6P5x4iTTULafZlxs4+Dw2SDj0GCSsIOanOJtra8PgnAfvOPWSBwst9Dvy8zTgc5PW2HdpANptM4OjyMzM2biNXX47E5c3Dp449zj9sSCcduyhQkDBIRoP/yuvVldaP2EOTJYKVSy13u+7EKBFZNQqW+jtG+jMx57LGS9luOoI56Gkyn0XfuHMaVWeWZmzdx9oMPcuszN2+i79w5AIhcoPAlSAghUgAyAJqklDsN1u+QUr4khOiQUu72voThYfTldeLLqgaeuupqjI6Poy2RwBw4HyCCPhmsVPFksuyyWwUCJ/s9in3Ox5culbxvO4Lc9HR0eDj3GzMzfu8ejg4PRy5IeN5xLYRoAgAp5XEAGfWxTocQ4kMAaU8LF0JGX171y2qXGnjaEgn89O//Hm2JBA6fPYtLn35abnHzhGkymJesAoFZ04+dJqFin+PlvIugdmhnbt50dLsw8WN003PI1iKAbBBoMdhmhZRynhJIyILZl7KcL+vR4WGkmpuxZulSAMCapUuxYvFi/N+PP7a9TyNhmwzmFatA4OSwV6N9lVIeNwWtvyumu/1nuduFiR9BIgbgmubxTINtmoQQLUKILo/KFFpmX8pyvqx11dXY0NqKOiXFcF1NDTa0tjp+jwAnr4qjxCoQODnsVd1XzbRpptuUM+/CCUGpTbQlEqiuqrLcprqqCm2JhEcl8k4gO67VfgohRKsQokVfoxBCdADoAICH5szxoYTB0ZZI5PVJAOV/WUfHx9F77BjWLF2KupoajN65g32nTmF8bMyJIuc4ORooSgp1gDvR76F9reGjR3Hn9u1J64QQZc27KJf2Nql+90+o/Qwc3eSNDIAG5e8YgM+0K5VObUgp+5R1k4660pm9GwDmL1gg3Sxs0Bl9ecv9sqp9EACwobUV+06dwpFz5/D1BQscKbPKqdFAUeRkICjErHlPShmYzyIIo56S8Xgkg0AhfgSJgwAWKX/HARwHACFETEqZQbafQu2wngeg1/MShozTX95kPI4bf/gDfjk0hJMXL2J8bAxfX7AAix9+2LHXUHl5MqTJrFJu2Gn2c2NIs3bUk9+BohJ5HiSklENCiEVCiBYAGSnlkLLqBICksr5DCHENwIea9eSR31++jMdmzcKzHo6PJ39YjSQrtdnPzSHNTAroH1/6JIzmPkgpk1br6T43J8+FOd2GnatYq+dEaaKfGauRZE5MztNO9HPieLI24b1AdlyTObcmz2mFNUCUehVr9RwAkZvoZ8TJfFNWQ5qdqGWwNuEPBomQsZo8V26QCMpwQzvspKsoNJnPqfQXXir1at3JEWZWAcfsWJ/56U8BlBgoAjgjO8qYKjxk3Jg8B2QDRJhvOWpnYp7Vc8I40c/O/SCcnHdhNb/DagRVqfesCOt3NKxYkwgJtR/CTBRnepbCTrNJoed4kfbbSXaT/zk1wsxqSLNV3ikva2heJcOMEgaJEND3Q+i5OdNTbb6orqvD+OhoYDtv7TSbFHpO2Cb6BaH2YxZwjI61lp0yltqJ7UV/XhSxuSkErDJQPlBbi6lVVThw+jS29vdjMF1aTkT17nJGVfj04CBOHzqET2fMwP88cACfzpiB04cOuXY7y3KU2myiBr974+MQQkx6jtN3ffNCkNOcqMdTPdZ6pZbRThJAN5JhVgLWJELAqr9h/O7dsq+MzNp4h48exe1HHsHOvXuxc+9eAEDX2rWB7bwtttlEP9JGSpmXG6nU/QVF0NOcqMfSqTKWOtrJrf68qGNNIgTM+huEEK5eGVXX1WH7nj15y7bv2YPqujpH9u+XqKYoN6v9AED/1q14+8UX0b91q681QT9raG4kw6wErEmEgFkSP7MmqGKvjApV1cdHR7Fp3bq8ZZvWrcOs0dGi9h9UQWi7d4u+9hPEGzs5XUMr9iZFbiTDrASsSYRAMh5Hqrk5d8UTq6/Pe6xXypWR1XDCRFsbpn30EbrWrsXExAS61q7FtI8+CkzzhV1Ba7tPDw66dqUf1VqTqpRh22a/I3ZaW2NNIiTMkvi5eWWkHdL4+qpVmDU6isTKlaFqpzcSpLZ7t6/0o1xrsqNSM7mWg0EixNxIE64Xts7bYgQpRbnduQ3FcjLtRtRwzkRxGCRCjldG9vgR/IxSZrh9pR+kWlOQcM5E8dgnUaHCnKcpjMxSZtQ+8IDh9k5d6YdxvocXOGeieKxJVCCn0oFXQiptp5g1K1VNnYqq6mpXr/Sj2GRopNhRTgDnTJSCNYkK5USAKDWZXCUzaz66c/s2r/QdUGpySs6ZKB5rEmSL2x2uUWPVgRy1K/0w1DA5Z6J4rEmQLRxaWRqrNNpREpYaJudMFI81CbKFQytLE6Rht04xqjGEqYbJkYHFYZAgWzi0snRRalYymwToZCpwCoaigoQQohHAswD6pJRXhBDtUsp+uy8qhEgByABoklLuLHU9Ocfu/SKieGVMxTOrMQghIKWctD1rmOFVbE0iBeA4gBVCiGMAWgHYChJCiCYAkFIeF0LEhRBNUsqhYteTc9SrweZUCq0bNuBYb29JKSGidGVMpbG6HanbQ3rJW8V2XB+XUl6QUr4CYCaAchrynkO2lgAAaQAtJa4nhwwfPYrmVApL16wBACxdswbNqVRkkr+Re6ySJHJIb7QUW5PIqE1MUsoTZneXKlIMwDXN45klrieHVNfVoXXDhtzjGuXxxZMnfSwVhYFVn1TYa5hGOZ0Ad3OkBVnBICGEeFBKOQJgRF0mpTzhaqkKEEJ0AOgAgIfmzPGzKI7xI9nY+OgojvX2YumaNaipq8OdsTGc2rsXt27ccPV1Kfyi2Cc1mE7j5+fP49bYWG5Z5uZNHDxzBkII3JuYyC2rpDxPxdQk/kEIcVBKOSyEWAhASinLSXCSAdCg/B0D8FmJ6yGl3A1gNwDMX7Bgci9ZyPiVbCzR1oYzBw8CAFo3bEBNbS0A4O6dO0gPDob6B0/uC3uNQUv/G9SakBLQdcareZ4qIUgU0ydxHkBcqVFcwP0TuF0Hcb9PI45shziEEDGr9VHmV7KxeDKJGuVWpOPK1dPSNWuweOVK9ktQRTH6DRZSKXmeiqlJxJG9ut+pDIU9BmDA7gtKKYeEEIuEEC0AMpqRSycAJC3WR5afycamPfgg+yWoLGFIw1GInd9apeR5KiZIpKWURwD8CACEEO3lvqjSXKRflrRaH2Wx+nrDL6kXX8JJ/RKjozi1bx/GQ34fa/JGEO+hbYfZbxAABAB9m3Yl5XkqGCSklEeEEHOVSXQLAczzoFwVxY1kY9qOcCEE/sv06RgdH0dbIgFtV786SgXI9kuc2rcvN3eCqJAwpeGwYvQbBLK/w3sTE5MmCC6qoJQeRQ2BlVJeUf6/AOCCmwWqRE7fhlTbCScnJtA4NoY93/8+1m3ahEOnT+Ov583LpVXWjlK5ePIkxkdHOa6dihakRI/vX71q+7lmv0H1sd6ljz+2/Vphw9xNAeFksjFtJ1x8bAx7d+7E3p3Z7CZru7rw69/9Do/NmpUXKBgUyI6gJHp8/+rVom42ZMXoN3jg9GnDbSul0xpgqvBIUr/AD8WlfFBfAAASCklEQVRi2LN9e966Pdu3o165ZWY5V17kjfTgIPq3bsXbL76I/q1bA5dyO+op0HlzItYkIknthPtDJoN1mzblrVu3aRNGZ83CFx99lPe5DrgwdApHaVKd0YRW3pyINYlIakskUF1VBQBI19ZibVcXJiYmsLarCx9Nm1ZRX/Aws+oUDpJ4Mon2zZvxtVdfRfvmzaENEH3nzuVq4doJrZV+cyLWJCJI3wl35YEHsOr11zE6axZWVlDOmbALUqdw1FlNaN3c3l7RvxkGiYjiXbfCLyidwpXAzwmtQcfmpgrHzuvginqnsFOc+A6zg9ocaxIe8iPTqxV2XgdblDqF3aIGiHKHv7KD2hyDhEf8yvQadFHI++MmzmEprNwAATg/oTVKGCQ8YtUx5vcX8f2rV3MT67wUhiGeVDnYj2eMfRIeCWrHmHoV5kffRFiGeFLwvH/1KvvTPMKahEf8zPRaiF99E2Ea4slmseBxopmJCmOQ8Ag7xiYLyxBPNou5g4E3HBgkPBKGjjGv+ybUNOX6Jqc5jz3mWRmKEZV02E5x4uReTuBlM5O3GCQ8FOSOsVzfxOXLngWKeDKJT65cwQdnz+YtT58/jy/MnevKCdjOCS5MzWJuc6pWZTfwOjXktZCgDVf3E4METeJljeLjS5cmLSvnKt0qCNg9wYWlWcwLTtWqygm8XgQIDle/j6ObKI/Xo52cvEpXg4D6XDUIqOm17Y6m4szn+5z6vMwCrFXg9eo7aTVcvRKxJkGTeDnaycmr9EJXuXZPcJz5fF85n5e2llf7wAMQU6ZATkzk1hcTeL0Y0RTU4ep+YZAgU+qVm52mp2Lb/o06r+1epRcKAuWc4DjzOcvu56Vv6hu7dQtTqqpQPW0a7ty+bfkd8aofQhXk4ep+8KW5SQiREkK0CCG6TNbvUP7v8LZkpPrio4/a/lEaNfucPnAA544cmbRtPJlEcyqVO1HXx2K277FdqAmDzUbls/t5GdXyJu7dQ3VtbVH3ofByToT2fiyqSh6u7nlNQgjRBABSyuNCiLgQoklKOaTbrEMIkQKwwevy0WSldmQbnRAA4IOzZw1HLTl1lV7oKpfNRs6w83nZberzY7hrKcPVK2EUlB/NTc8BOKb8nQbQAkAfJFZIKY97WioypPZPlBIorH74bs4tKCYIsNnIO2qTY3VdHYQQkFJO2qZQR3WxNQinT9bFDFevlFFQfgSJGIBrmsczDbZpEkIAQJOUcqd+pdIM1QEAD82Z40YZSaPUjmyztn/A/bkFDALBkB4cxOlDh3D7kUewfc8ebFq3DrXpNKZkf9cArJv6SqlB+HWyDnLSTicFcgislHKnUpOYKYRoMVi/W0q5SEq5aEZDgw8lrEzFJlWzauOvxLkFUZUeHET/1q14+8UX0b91a26oMZCtzd1+5BHs3LsXU6ZMwc69ezEWj0O5+LPsyyi1o9qvIauVMgrKlZqESYdzWjnxZwCoZ/YYgM90z00BgJSyT1kXnZDsA6eq4eoPtpimJ7OZ1Owkjo5CExOr6+qwfc8e7Ny7N/ec7Xv24PVVq/D0iy8a7lN7AVJKR7VfJ+tKGQXlSpCQUu62WH0QwCLl7ziA4wAghIhJKTPI9lOklfXzAPS6UcZK4EY1vNimp+Znn8UX5s4NRCcxE8k5r9CclPHRUWxaty5v/aZ16zBrdNRyv3ZGMfl1sq6UpJ2e90lIKYeEEIuUZqSMZmTTCQBJZX2HEOIagA8NRj5RkdxsMy1mDkUQ+geYwdUdhfqcEm1tOH3oELrWrs31SUz76CMkVq7M69AeHx1Foq0No488Yrssfp2sw5C00wm+TKYzqmlIKZNW66l0blXDS2l68hszuLqj0MRE7Uiz11etwqzRUSRWrgQAnOvrQ3MqhdYNG3CstxdnDh/GH//N3+C/P/OMrbL4ebIOctJOp3DGdUgV09dgVQ13oq9COzwWsDczu1h2m4yYwdUdxcy8NqpJ9m/diuZUCkvXrAGA3P9Dv/yl7SABVMbJ2i+BHN1E1tS+BjUAqH0Ng+l03nZmM0cfmzOnqOcXQzsz262JT4US91mxk0iOCrM787q6rg6tGzagpq4OAFCjPK7VzYSn4GBNIoSK7Wswq4a70VfhZlLAQk1GVrUMJ3NDUT47fU63bt3Csd5eLF2zBjV1dbgzOopT+/ZhzGCGPgUDg0QIldLXYFQNP3D6tOnzX3z77bLadN1oerJqMirUMc1UHMEy5y/+IpfDq3XDBpzatw/njhzBgq9/3eeSkRkGiRAqd8if2fNVdofKutWhbdVJWkzHdBBGWVU69eJh5uOPI/bQQxh65x1cPHkSY+PjWPD1r+PhxYt9LiGZYZAIoXKH/Bk9X6+c5ienO7StmoxOHzhg+Bx2TAeH0QxqBoXwYMd1CCXjcaSam3M1h1h9PVLNzUWf0PXPN1POUFknO7StOknZMR1c2jQuXqb6JmexJhFS5Q750z5/a3+/azNWnapVaJuM1I7qiydP4u6dO7bucEbuOnLpEn79u9+h/oEHMDo+jrapUzlENaQYJMj1Gav6vgqV3Tve6bOL1l25gtoi7nDmpFLmbVRCWhDt53rp009x7MoVpJqbsaG1Fb3HjkUyhXalYJAgz2asapsc7NYutNlF1eRxXWvX4o+uX8dzL7/saHnNpAcHcfbQIUwoQfVmJoOzhw4BmJzqoxLSguiblN567z2kmpuxZulSAMj9H7UU2pWCfRIEIBsoNre349WvfQ2b29td/zHb7bNQs4tqbd+zB9XK5CwvnP/5z3MBQjVx7x7O//znk7a1Gn0VdmZ9DnXV1djQ2oq6mprs45qa7GNOmAsl1iTIV7lAoUzEK1SrsJtd1Eljt24VvTxqaUH0AV3fIT2YTuM/b9xA77FjWLN0KepqajB65w72nTqFUU6YCyUGiYgI+7129R3cgHHAsMouGkSFEuGFRTH3elDTxUgpcxM2N7S2Yt+pU+g7dw6p5mZPykrOYpCIAC9v3+hmMNKffN7XpfmYP3u2aXZRL9v3a5ROcqPlemFOC1Ko1qCnT/dy4PRpnLx4Ef954wb+dsmSUF200H0MEhHg1b12vb6XsFFHtxoo/Oz0feIrX8GZgwfzht2KKVPwxFe+MmnbYtKCGN1fwc/35+Qd4v6g1KIYIMKLQSICvLp9o583fvcyLXkhpeaDsgpqRkN6T5uMlPJCOZPfKuV2npWGQSICvPpx+n3jdyfnW5TLqdqM2ZBer26KVGqTkpVKuZ1npWGQiACvfpxBuVI0m2+hFeS75WmpQ3rVAAFkh/S+vmqV469lNtTYqZQZlXI7z0rDIBEBXv04zRIDPjZnjqOvUwqzE5xRp3cQuT2k18maQjF4h7jo8S1ICCGapJRDJutSADIAmqSUO70tWTh58eNMxuO48sknOPvBB3nLzyt3tLv08ceBuYIMS23DqSG9btcSqHL5EiSEEC0AdgCY1OgqhGgCACnlcSFE3CqYkPcuffzxpGXj9+7lBQ63Rz2VyuhEaRY4VPNnz8bZf/93vPOv/4rq2lqMj43hmT/5Eyx++GFHy1bKkN5CM9MZEMgNvgQJJQBcM1n9HIBjyt9pAC0AGCQCothOaq9GPdlldUL9/eXLOHLpEn45MoIVixfnktT9kxL43AgUhTqpmXKb/BLEPokYAG0AmelXQWiyQne10/Jq1JPTvvjoo3jrvfewYvHiSUnqDv72t5heW+tbuYi8FsQgUZAQogNABwA85GOnaaXQzrKeVlODqilTcE8zkcxMmMfHq0nqco+VJHUnL17kyZoqiitBQjmJ66WllMeLeHoGQIPydwzAZ/oNpJS7AewGgPkLFki75aTC9LOsb9+5gylC4IHaWtwaG0Osvh6PzZmD8+l0pMbHj46PV0SSurDn/CL3uRIklJN4SYQQMSllBsBBAIuUxXEAxQQWconRLOsJKVEzdSr+t2YEztwvfCFSJxt1uC8Q3SR1XqdZoXDya3RTCsAiIURKStmnLD4BICmlHBJCLFJGQGU4sslfxc6yjtr4eO3ck5MXL2J0fLyk+4iHgZ9pVig8/Brd1AegT7csqfm75JoIuSMos6z9ELXAp1foAoBNUQSEtOOavON3Ph6eqNxjdQHApihSMUiQJT/z8Xh1oqrUQGR1AcCmKFIxSFBBfjW7eHGiquQrZqsLAPXOcnphnftC9jFIUGB5kZq80q+YzS4AKrkvivIxSFBgeXGi8vseGV4ptUnN774oCo4pfheAyExbIoHqqqq8ZU6fqMwCTpSumNUmNTXwqU1qg0r2XiPJeByp5ubccYjV10duCDAVhzUJCiwvOs0r4YrZbpNa1IcAU3EYJCjQ3D5RVcLd1CqlSY3cwSBBFS/qV8zshKZyMEgQRZS2s1qvUJNapc4dockYJIgiSD//Q6vQSb+S547QZBzdRBRBRp3VQDZAbG5vtzzZW3V0U+VhkCCKoHI6q9nRTVpsbiJyUFDa8svprGZHN2kxSBDp2D3RB6ktv5z5H5Uwd4SKxyBBpFHOiT5IeaDKmf9RCXNHqHgMEkQa5Zzog9aWX878j6jPHaHiseOaSKOcE30l5IGiysMgQaRRzonei4SERF5jkCDSKOdEz8ypFEW+9UkIIZqklEMm63ZIKV8SQnRIKXd7XTaqXOV22rItn6LGlyAhhGgBsANA0mSTDiFECsAG70pFlMUTPdF9vgQJKeVxIcQ1i01WSCmPe1YgIiIyFNQhsE1CCABoklLu9LswRESVKpAd11LKnUpNYqbSNEVERD5wpSYhhOgwWJwupglJ6YuAlLIPwGcAJjUOK/vvAICH5swpr7BERGTKlSBhZ0SSECImpcwASCv/AGAegF6T/e8GgPkLFsgyikpERBZ8aW5SaguL1FqD4gQAKMNiVyrrPjQbJktERO7za3RTH4A+3bKk5m/OjSAiCoBAdlwTEVEwMEgQEZEpBgkiIjLFIEFERKYYJIiIyBSDBBERmWKQICIiUwwSRERkikGCiIhMMUgQEZEpBgkiIjLFIEFERKYYJIiIyBSDBBERmWKQICIiUwwSRERkikGCiIhMCSnDfYtoIcRVAP/mczFmAfjU5zLYwXJ7K4zlDmOZAZa7kD+WUs4uZsPQB4kgEEKcl1Iu8rscpWK5vRXGcoexzADL7SQ2NxERkSkGCSIiMsUg4YzdfhfAJpbbW2EsdxjLDLDcjmGfBBGFnhCiSUo5ZLIuBSADoElKudPbkpkrUOYdUsqXhBAdUkpfAwdrEg4QQjRZrNuh/N/hXYmKU6DcKSFEixCiy8syRUGhYxfUY1tEuQP5XRZCtAD4kcm6JgCQUh4HkLH6znvJqsyKDiHEhwDSHhXJFINEmcL0YWuF8YcFBP9EVujYBfXYFlmuQH6XlTJfM1n9HLK1CCBb7hZPClVAgTIDwAop5TxlO18xSJQpTB+2Vhh/WCE5kRU6doE8tiiuXIH8LhcQQ/73fKZfBSlRU1BqmwwS7gvMh12CoP6wwnAiK3TsgnpsiylXGL/LoSSl3Kl8h2cqtX7fMEi4LEgfdgTwROajkH6XMwAalL9jAD7zsSxFUZpUU8rDzwDE/SzPVD9fPAxM2rbTxVypqh+0lLIPHn/Y5ZQbIfxhqdTRK0KIViFEiw81ikLHLqjH1rJcfn6X7RBCxKSUGQAHAagzmOMAAttUpilzGvebS+cB6PWvVAwSBdkZfhaED7vMcvv2wyoQ3MJwIjM8dkE4tgUUKnegTlxayue+SAiRUj57ADgBICmlHBJCLFJqPhmzIadeK6LMHUKIawA+9LvMnCdRJuXD/hGAb6ofthBiUEqZVP7uQLaJJB6wMdrFlDuNbLkDMcFH6aheJKXcrTQnHVd+UDEppdqRnVb+7gXQ68cPzOjYBf3YAkWXO3DfZXIXgwSFCk9kRN5ikCAiIlMc3URERKYYJIiIyBSDBBERmWKQICIiUwwSRERkipPpiByizD2JIztE9wkAP1AmohGFFmsSRA4QQsSVSYlqUDioBgghRFOIch0R5WGQIHKAlFJNWZGEMhNcs/o5ZNOIEIUOgwSRAzT3tohrUoSoDvpRJiInsE+CyBktQog4gGNK05LVjaiIQoNpOYhcpuSTmgd2ZFMIMUgQEZEp9kkQEZEpBgkiIjLFIEFERKYYJIiIyBSDBBERmWKQICIiUwwSRERkikGCiIhM/X9s66pK+NfXfwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cfae58d358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "plot_contour_2d_classification(circle_X, circle_y, lambda x: circle_svm_mapped_classifier(feature_map(x)),\n",
    "                               delta, \"SVM classifier with standard kernel operating on mapped features\",\n",
    "                               lambda: add_important_vectors(circle_svm_mapped, [0, 1]))\n",
    "print(\"Accuracy: \" + str(accuracy(circle_y, circle_svm_mapped_classifier(circle_mapped_X))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, our idea worked out well. The support vector classifier is able to separate the 3 dimensional mapped data set based on the length of the vectors, because the classes can be linearly separated just by looking at the length of the vector. Sometimes, though, we may not achieve an accuracy of 100%. This is due to our regularization paremter *C*. If we were to use a hard margin classifier, we would achieve an accuracy of 100%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.7\n",
    "Another approach to introduce non-linearity to the support vector classifier are *kernels*, which are functions that replace the dot product in the algorithm and - broadly speaking - map the data into a higher dimensional space and calculate the dot product there. By introducing the non-lineartiy this way, we are capable of implicitly mapping our data into much higher dimensions than a standard feature map would be able to (even spaces of infinite dimensions) where it should be easier to separate the classes linearly. However, this approach obviously comes with the downside that general kernel functions, like the **gaussian kernel** (also called **RBF**) are not capable of using very specific knowledge about the data as we did with our feature map. Yet in most circumstances, no knowledge like this is available, so kernel functions generally promise far better performance than conventional feature maps.\n",
    "\n",
    "Fortunately, we have designed our SMO algorithm to work with kernel functions from the start. So all we to do now, is to write a function for the gaussian kernel and plug it into the algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_gaussian_kernel(sigma):\n",
    "    \"\"\"Creates a gaussian kernel function given a sigma\n",
    "    :param sigma: sigma for the kernel\n",
    "    \n",
    "    :returns: function. gaussian kernel for specified sigma, returns Matrix M with M_ij = K(X_i, Y_j). Returns an array if\n",
    "              X is a single example.\n",
    "    \"\"\"\n",
    "    def f(t, X):\n",
    "        if len(np.shape(X)) == 1:\n",
    "            X = X.reshape((1, len(X)))\n",
    "        if len(np.shape(t)) == 1:\n",
    "            t = t.reshape((1, len(t)))\n",
    "        result = np.exp(-1 / (2*sigma**2) * dist.cdist(t, X)**2)\n",
    "        return result if len(np.shape(t)) > 1 else result.reshape((len(result),))\n",
    "    return f        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "sigma = 1\n",
    "gaussian = create_gaussian_kernel(sigma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "circle_svm_gaussian = SMO(circle_X, circle_y, circle_C, max_iter, gaussian)\n",
    "circle_svm_gaussian_classifier = create_classifier(circle_svm_gaussian, classes, borders)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAERCAYAAACO6FuTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X1sVOedL/DvY2Njx9tkeOtNacXFkzarpDfawUP2OjRdNhT/YaFIjZk4haTFa7q2tCW325I4roBd6YZojYObZYNWMk0s07QQHGNacVmrgtjLJgauFoOlXSdqooxZbpW2gZghErGNwc/9Y84ZzozPmdfzPt+PhPCcM3PmmbfzO8/b7xFSShAREekpcboARETkXgwSRERkiEGCiIgMMUgQEZEhBgkiIjLEIFEkhBB7hBCjyr+Isu2kEKIm5X5tQoj1yt9S/Vuzv1sIcTLDcwWFEHvyLOd6IUS3pizdyra2fI6XzfOkbM+77GYwKleWjzW97OmOKYRoEUK8pX6fnJT6PSbzLHC6AGQ95UQfkFKGldsBZVc3gKcAXNDcvU5K2an8HQXwJIBTmv1BK8sqpTyleb6HpZRPKn+fMnhIwc8jhIhIKfvNPH6+Ul6/29VpPh/HKN/vPQDCTpfFj1iTKA4BADH1hpQypvzfDyBxFSiECGrvp/y9WLO/BskBxWqBzHcxxVM2PY/f2PX5pKUE1kmny+FXDBJFQAkG65WmppaU3Rc0VfUI4rULrX/X7H8KwBG951CbobRNUUKIgNIccVJ9XqU55S3l/jU6t4NK01gbgNWa7XuUx6vHU5vMgsp93lKCnLZMJzXl+Ej5u0Y5vvo8e5T3pltTw6pRbo9qtqW+XrUc3UqzmN5rDaY0nUVyeP1GxzuZqWzqa1fun3i/Ut6rv9Q7Vur7m+b46ueT+rmor9fwc9F8D7qFENeyeT5ykJSS/4rkH4D1AE4C6NZsiwDYo/x9MuX+o4g3L6n730L86jH1fi0A2jS3E4/RHkv5vxtAjWZ76u1gankA1CDenNAGoEUti+b+1xBvTkt9vW3Ka25RniegbKvRex7N8dTn3QNgvcH72KYth8FrDarvtfK8ahDO6vUbHM+wbOpjlc+pJvX90r5XesdK8/7uSX2NKZ9PG4CI5vvUkuFzCWjel4j6WJ3X0qL3L11Z+M/8f+yTKCJSae8WQoxqtvUrV67/gHgfROpjouoVLoB/Nzh0GPNrIAASV5xLcKcvYw+APcqVa6vO7XTuA7BYCBEGENBcSfdJpQktRb9yzIBSvkYofS56V7ca6vvwKfSbVCaV1xaApplD57Xqyfr1GxwvU9naEG8SjCHl/QJwN5T3SgixWOdYRu9vJg8DOKD8fUF5Tadg/Lm0KPcB4s2Zut87zTHJQQwSRUBpLooqJwe9E5j6w37L4BDq/hcM9o8ifiWa1F+hnOSiykk5AiR+/E+KeGdjq5TyBe1tGAQbzfNMSk0ns3Ky0zsRqQEuqJThghCi1ei+eahD/OT4glKOea81xX3K/qxefxbHM3JAKdNbyr/E+yXm9zmlMnp/M/l3xD//fuV/9WLC6LmWKO8DAISllPOCgVJW3dct7wysIBswSBSPtzQ/+NQRKd2INy8YXckfUfZH9a4upZQHxJ2hsTHcCSanAPxMCFGn3lc5+am3X0i9ne4FKM/zlnqyl9mNrJkE8JHy92LEm9vmUcqeqSajpZ5w9wghXoDOa1Xer9VKO31QeZ5sX/+842UpplwMqAH/KU1wtOL9hRLI3hJC/ATxwPZkhppat9LPMgmDiwIliGQVDJQgulq4aJSanwilPY+IsqScgF9QgsB6AE+mCbBEnsYgQZQjpflOGxS6pZR2Dg0msg2DBBERGeI8CSIiMsQgQUREhhgkiIjIkOeHwAYWL5bLv/xlp4tBROQZ7//nf16VUi7L5r6eDxLLv/xl/PxXv3K6GEREnvHwV7/6X9nel81NRERkiEGCiIgMMUgQEZEhz/dJEJELzM2h9OZNiLk5gBN03UEIyJIS3C4vB0ryrw8wSBBRwUpv3sQXFy/GPYEAhBBOF4cQXyvoeiyGTyYncbuiIu/jsLmJiAom5uYYIFxGCBH/TObmCjoOgwQRFU5KVwSIXTt24PENG/DM5s14pasLExMT2LVjhyXP9ey2bZYc10xCiIKb/xgkiMgXnt22DdXBII6fOIFfHDqEv9q61ZLnicVieHbbNhw7etSS47sN+ySIyHbvDQ3hnd5efHblCu5etgzfbGrCg+vW5X28WCyGfx0awqv79ye2BQIBXLt2zYziJgkEAnh1/35cvFAc2eEZJIjIVu8NDeE3+/bh1swMAOCzTz7Bb/btA4C8A8XFCxfwl2keG4vFsO1v/gbXr1/HEw0NaN66FcNDQ3j9tdewaPFibN26FdeuXUu6HVq1Kq+y+A2DBBHZ6p3e3kSAUN2amcE7vb0F1SbSCQQC+MWhQwCAR9esQfPWrRgYGMBzzz+fCAbPbtuWdJviGCSIyFafXbmS0/ZsrKqpwf/K0JH8SlcXJicncWliAgDw4+3bsWvHDly/fh3/tH//vNvV1dWJx+7asQOx69cRrK7Gj7Zvz7ucXsSOayKy1d3L9JOPGm3PRiAQwF+uW4ee119PbIvFYom/X+nqwsrqarz40ktYtGgRAKC6uhq/OHQIP96+HT2vvTbvttaLL72EV/fvL7oAATgYJJR1go327VH+b7GvRERkh282NWHBwoVJ2xYsXIhvNjUVdNxX9+/HRDSKxzdswOMbNqDr5ZcT+x5btw5de/cmDVt9pasLj2/YgJ92dWFjJDLvdjrPbtuGSxMTeHbbNkwoNRO/cmSNayHEegB7pJRhg/3XAEwCaJVSnkp3rAcfekgyVTiRsxbcuIGv3X9/1vc3e3QTGfvwgw9wq6oqadvDX/3qqJRydTaPd6RPQkp5SggxmeYuT2YKDkTkXQ+uW8eg4BFu7biuUWZv1kgpO50uDBFRsXJlx7WUslOpSSxRmqaIiMgBrgsSQoiIEELtNfoUQFDnPi1CiPNCiPPXJtO1WhERUSFcEySEEAHlzygAtT/iPgDnU+8rpTwgpVwtpVy9aPFiu4pIRFR0HAkSSk1htabGAABvA4CU8gKARmXfR8ptIqKMrM4C64XMr2ZzanRTP4D+lG1hzd8HbC8UEXnas9u2YdWqVXjxpZcAxCfTmZXgLxaLYdfOnTh29GhSEsFi4NbRTUTkY8Pj4zh4+jSufvYZlt59N7asXYvHvv71vI9ndRbYYsv8qsUgQUS2Gh4fx6uDg5i5dQsAcOWzz/Dq4CAA5B0omAXWOgwSRDkajUYxODaG2I0bCFRVoT4UQjg4bxAeGTh4+nQiQKhmbt3CwdOnC6pNpMMssPlzzegmIi8YjUbRf+4cYjduAABiN26g/9w5jEajDpfMO65+9llO27OxqqYG/zo0lPY+r3R1YdeOHUlZYPe+/DIe37ABExMT825THIMEUQ4Gx8Ywe/t20rbZ27cxODbmUIm8Z+ndd+e0PRtWZ4EtZgwSRDlQaxDZbqf5tqxdi4ULklu6Fy5YgC1r1xZ0XKuzwBZT5lctR7LAmolZYMlOuwcGdANCoKoKOxsaHCiRO+SaBdbs0U1kzJNZYIm8qj4UQv+5c0lNTmWlpagPhRwslfc89vWvMyh4BIMEUQ7UUUwc3UTFgkGCbOGnYaPhYNCw7H56nUQAgwTZQB02qjbRxG7cwJEzZ/Dr8+fx+cyMb06meq+z/9w5ALD0tTEwkZU4uokspzdsdE5KfD4zA8A/cw2cGB7LeRtkNQYJslw2w0P9MNfAieGxnLdxx64dO/DomjV4dM0aHBsYcLQsj2/YgLGLF5O2vdLVheEME/7ciEGCLBdIGX5nxOtzDYxeZ7avPx+ctxE3PDSE2PXrePfMGbx75gwes2n9bKNg1Lx1K472JyW6xtDQUNblcjrIaTFIkOXqQyGUlZZmvJ+VJ1M76L1Oq4fHOhGYzKSsZV+wWCyGwD33JG4HAoE09zZPaiBQPdHQgF8dO5a4PTExgXs05cv3uE5gkCDLhYNBRGprEyeuyvJylJYkf/X8MNcg9XUGqqoQqa21tBM5m8A0Go1i98AAnnvjDeweGHBNf8WC0lLctXAhFmRxAZHJEw0NGB4exqNr1iSl5phQZkgD8eaeYwMDmJiYwKNr1uDZbdvw0IMPYmJiQncbADyzeTMe37AhcQz1eM9s3oxnt23Dvw4P49lt25JSgKj+LBRKNDn9amAAW7///cQ+9bhqjeHZbdsSM8V37dgx77jpynHol79MlCe1icsMHN1EtkgdNurXETnphsda9XyA8bwNp0ZcZbKgtBTlSmoO9f9bKX0ruXr3zBkMDw3hp11duHjxYtrFgVauXIlX9+/HsYEB9Lz2Gpq///152xYvXoyNkQieaGiIb3v9dTy2bh2OHT2K/3z/fQQCAVyamDB8no2RCI729yO0ahWGhobwo+3bAcSD1bpvfQvNW7fimc2bce3aNQSrq5OOMzY2lrj9ipImxKgcu3butDR7LYMEOcLuk6mfpXsv03VsO/X+CyGwsKxs3u3bc3MoNE3QY+vW4bF16/DomjVp77do8WIA8RqIWvNI3XbPPffgr7ZuBQCEVq3Crh078Ni6dXhi48asmrOeaGjA3+3cie3PP4+V1dWJ7dGJCVybnMTFixdx/fp1nDt7Fn/zgx8YHmd0dDRtOX68fTt27diB69ev45/270e15rnMwOYmIh8walJyY8e2lBIzs7OJgJB6Ox9jFy8mmmaMku9FNduvTU4CiHd4qyfw1G3hcDgxGml4aAjhcHyF5UAOfQt/Fgph186daNDk9Vq1ahU2RiJ4df9+HD9xArWPPJJ21FOmclidvZY1CSKPS9ekFKiqMkxI6CS1aWlhWRlu3rpVcFMTAHz36acTy5W+8ctfAoifQC9euJDI4Kq6dOkSntm8GZcuXcL/+Zd/wbVr1+ZtCwQCeGbzZnTt3YuVK1fiF4cO6QagxzdsMLyC3/r97+O7Tz+d1JSkNjOptZVfHDqU6JNQb2uP+6Pt29OW45WuLgwpQeTF3bsLfh9TOZYFVghRI6XUXTBWCBEBEANQI6XsTHccZoGlYpcuM61RQkKzO9RzzQKrEkIU3MSUq4mJCfy0qyvpxK23zS8KzQLrSHOTEGI9gJ8Z7KsBACnlKQAx9TYR6UvXpOTEiKtceH2pgmLgSHOTlPKUEGLSYPdTAE4qf0cBrAegW+MgosxNShwkkKw6ZSSR0TaKc2PHdQCANoAscaogRF7gxCQ+Kh7suCbyOK5xQVZyY5CIAVis/B0A8KmDZSHyBDYpkVVc09wkhFBnpxwBoH7bgwBO6dy3RQhxXghxXh3bTETmUude7D1+3FXpPPT8SWXlvLkG6rDSfAwPDSVSYGTip4yvepwa3RQBsFr5X/U2AKjDYpURUDG9YbJSygNSytVSytXqLEkiMs9oNIq+kREsunoVh3/4Qyy6ehV9IyOuDRTV1dUYSMmceslgUl02Hlu3LuuObD9lfNXjSJCQUvZLKRdJKfs128Kavw9IKU9JKQ84UT6iYjc4NoYVU1Po7exESUkJejs7sWJqyrXrVNwTCEDbqjB28SJCNnXc+ynjqx7XNDcRkXtUlJWhp6MjaVtPRwcqNDmXCnHr1i0cPnQIUkocPnQIt27dKviY4XA40exztL8fGyN3GipisVgik2rP668nZVGdmJhIyrL6SlcXJiYmsGvHDkxMTCS2P7pmjW62V8A/GV/1MEgQ0TzTs7Nobm9P2tbc3o7p2VlTjv9WXx/+eutWfOGuu/DXW7firb6+go/57YaGxFX5xMREUlK9QCCAXxw6hOMnTiQS+h07ehT7//mfcWliAuFwGMdPnMC1yclEtlatV/fvx2OPPYaLF/SnbKkZX4HkpiY14+vxEydwtL8fPa+/jmB1NY6fOIHjJ07gxZdewqqaGry6fz8CgUAi4+vxEyewbt26eWU9e/Ysnnv+eby6f79lWV9TMUgQ0Tz1oRAuV1aiqa0Nc3NzaGprw+XKStPmXnxn06a0t/NRXV2NsbExTCgn/VSvdHVh144dib4KNYvqokWLAMRrG3p9nGqwWbx4sWFNQm1yisVi8zK+Dr39Np7dti2R8TVdX8Xo6Ghif2jVKgy9/XZSWX+8fTv2vvwyHt+wwTCRodncOASWyNV+/8EHlj/Hl/LIg2Qm7dyLTfv2YXrpUjSaOPfizcOH593etHlzwccNhULYtWMHXnzppaTtr3R1YWV1NX60fXui/0CbzXVoaAijo6PzHpcLo4yvixYtwhPKtp7XX8fw0JBhLUDN+PpEQ0PajK/DQ0Poee21gsqbLQYJIkUuJ/8Hly2zrBzvXbmSU1msCihWzr14srERQLwG8ebhw4nbhdoYieC7Tz+N6urqpKv+x9atw7Yf/CCRLTXVpYkJ3BMI6AaYbPkh46sex7LAmoVZYCkX6U6+Vp74rfLelStp99tVI8k3C6wbPLN5M1586SVUV1djeGgIAwMDvsrjVGgWWNYkyJeMgoEXA0E66V6PUY3E6aYst3nu+efx066uxO2tyipwFMcgQb6jnhj9FhBypff6tYGDwSIutGqVr2oOZmOQIM9LvVou9uCQjvrepNYyCg4YyuJBQojCjkOmklICBX4mDBLkSQwMhdG+X2YEDFlSguuxGO4JBBgoXEJKieuxGGRJYTMdGCTIMxgYrJH6Pr6X8j5nEzRul5fjk8lJXLl6FfD4YBjfEAKypAS3y8sLOgyDBLkWg4Iz8qpllJTgdkWF1UUjBzBIkKswMOQmOjqKscFB3IjFUBUIIFRfj6DObON8pQsYADu/iwGDBLmC9uTDwJCd6OgozvX347aST+lGLIZzSv4gMwOFal6zlBI0GCj8jbmbyHHaIasMENkbGxxMBAjV7dlZjA0O2vL86mf1+w8+sCVVCTmDNQmynZ+blKxu/tG6YZBszmi7FVKH1LJW4T8MEmQrP090s7v5pyoQ0A0IVYGAzr2zk2+QSwQL9ln4DpubyBZqk4Sfm5Tsbv4J1dejNGURoNKyMoTq6/M6nhrk1MCjBrno6GjWx1A/X21TFHkbgwRZzs+1By27m3+C4TBqI5FEzaEqEEBtJJJ3rcXsIMc+C39gcxNZpliCg8qK5h89ZvZ7aI9lpJAgxz4L72OQINMV63DWUH19Up8EkH/zj1EgMLPfI/VYRswIcql9Fl4MFqPRKAbHxhC7cQOBqio8sHw53v/448TtehMXZXITBgkfSP3yOvllzbX2oJ4MyyoqMDs9beloIKup5S70Kj9dIEjXJJTr8+gdS8/yBx7I6bjpPLhsmScz0Y5Go+g/dw6zt28DAGI3buDshx8m9sdu3ED/uXMA4LtA4UiQEEJEAMQA1EgpO3X275FSviCEaJFSHrC/hN6h9+U148uqBp6KsjJMz85mDDz51B6io6MY6evD1IoV6OjpQXtzM0b6+gBYMxrIDsFwuOCypwsEZvZ7ZPuYj99/P+djp+PFJqjBsbHEb8zI7O3bGBwb812QsL3jWghRAwBSylMAYurtFC1CiI8ARG0tnAfpfXnVL2u+1MBTHwrhzb/9W9SHQug/dw6jUf2PI9/JcGODg5hasQKdvb0oKSlBZ28vplassG0ymFulCwRGTT/5NAll+xirOt691LEdu3HD1Pt5iROjm55CvBYBxIPAep37PCmlvE8JJJSG0ZeykC/r4NgYIrW12LJ2LQBgy9q1iNTWzgs8hQ5rLauoQEdPT9K2jp4elBV5orh0gcDMYa96x8qlPGbwynDZQMryn4Xez0ucCBIBAJOa20t07lMjhFgvhGizqUyeZfSlLOTLWlFWhta6OlQoKYYrysvjt3VOKIV0TM9OT6O9uTlpW3tzM2anp/M+ph+kCwRmDntVj1VeWWl4n0LmXeTC7QMc6kMhlJWWpr1PWWkp6kMhm0pkH1fOk5BSdiq1iCVCiHk1DSFEixDivBDi/LXJSZ0jFA+9L2+hX9bp2Vl0nzyJ6Zs347dv3ozf1rSTm3HVF6qvR+Xly2hrasLc3BzamppQefmyLSclN8sUCILhMBp27sR39+5Fw86dBfWBBMNhlC1cqLtPCFHQvIt8uLXpKRwMIlJbm7j4ClRV4ZGvfS3pdqS21nf9EYAzHdcxAIuVvwMAPtXuVDq1IaXsV/bNe9eVzuwDAPDgQw8V9Qon6pfSzNFNah8EALTW1eHg6dPoP3cOkdpaU4e3akcD7du0CUunpxFqbPRsp7WZzOgAz5ZRn4OU0tbPwu0d2uFg0JdBIBMngsQRAKuVv4MATgGAECIgpYwh3k+h9pDeB6Db9hJ6jNlfXm3gGR4fx/TsLCK1tVh+6xYAc5sG7DwZ0nzpUm7k0xdhxkQ/7TBZtwWKYmR7kJBSXhBCrFaakWJSygvKrrcBhJX9LUKISQAfafaTjVIDT7HNni4W6UaS5drsZ+ZEPzVQkPMcmSehN/dBShlOt5/usHvynFcCRD5XsekeY2fab6ekG95qxuQ87US/XN/PB5ct8/QMbb/gjGuPsWrynBEvBYhcr2LTPQaArWm/nWJmvql08zvyrWV4dYa2nzBIeEy6yXNW1SbcHiCA/NJVZMp6alb6CzvlerVuZr6pdAFH773+wpIlOPPmmwAyBwoAbH5yiCuHwJIxKybPGXHjUEQj+aSrSPcYN6z6lqt81oMwc95Fuvkd2vdtTkpMVVfjh4cP4/OVKzHS15f1mhVe+k76BWsSHqH2Qxgxe6anOpPaK/JpNsn0GDvSfpsp3+R/Zo0wS5fgUJt3aiYYRGdvLzp7ewEAbU1NWdXQzOjMzjUnGbEm4QlqP4RRbcHsmZ7aq7Xo6CgGdu/G8b17MbB7d06rlNkpn3QV6R5j9qpvdnBD7cdoop/6fgbuvdexVCyj0Sj6Rkaw6OpVHP7hD7Ho6lX0jYwY5iSjOAYJD0iXgfKuhQuxoLQUw+Pj2D0wUPAXXttRrTZfhOrr8bdvvplov3ZjoMi12URtu789OwshxLzHmL3qmx3MTP5nNvX9vP7HPxaciiXfJqfBsTGsmJpCb2cnSkpK0NvZiRVTUwUlwywGbG7ygHT9DTMzM1gxNYWen/wEze3t6BsZAVDYSCe1mWlscBC1kQjWbtkCAIn/3dp5m22zSepIGyllUm6kXI/nFmZ2QltBfS9H+vrQ1tSUSA9fefkyQo2NWR2jkIl2FWVl6OnoQG/nndUJejo6sGnfvpyOU2xYk/AAo/4GIYSlV0ZlFRWoa21FudIUUK7c9nqWVrPXcnYLo9oPAAzs3o03nnvO8SbDYDiMbzQ2Yum1a/FULNeu4Rs5pmLJt69senYWze3tSdua29uTcpLRfKxJeICaS0nb5FRWWoolX/gCev7u70y7Mkqtxs9OT+NkdzfWbtmC8ooK3JyexumDBz2fpdUNbfdWSa39mDkL2qoy5ivXuRP1oRD6RkbQ1NaGno4ONLe343JlJRp9mLnVTAwSHmCUxG9wbEz/ymjp0pyfQ280k9p8AQB1ra04ffAgzvX3J65OvcrMCWRmsHJmt5nLnbpJPnMntL+jTfv2YXrpUjRydFNGDBIeYZTEz8orI+2QxvHhYcxOT7u+8zYbbmq7t/pK38+1pnwUaybXQjBIeJgdV0Ze67zNRrrx/Haz+krfbbUmN+GciewwSHgcr4zy40Tw02tWsvpK3021JjdR50ysmJpK1MLNGBnoRxzdVOTcuhKY3xilzFh411269zfrSt+L8z1y8eCyZXl9hzlnInusSVDeQwqLIZW2WYyalUoXLEBpWZmlV/p+bDLUyiddB+dMZI81CcpLPsnkiplR89HNqSlfX+m7FedMZI81CcqLX4dWWiVdB7LfrvS9UMPknInssSZBeeHQytx4MWFgPrxSwwwHg2j8xjdwbelSbNq3D9eWLkXjN77BTmsdrElQXji0MjduGnZrFr0ag5M1zFzzOXFkYHYYJCgvHFqZOz81KxlNAkwNECqra5iFJP6j9LIKEkKIagAbAfRLKS8JIRqklAP5PqkQIgIgBqBGStmZ634yx+8/+ACfjo9j4N13UVZRgdnp6ayvbv14ZUzZM6oxCCEgpZx3fztqmGYsSkTzZVuTiAA4BeBJIcRJAHUA8goSQogaAJBSnhJCBIUQNVLKC9nuJ/N8Oj6O//eb36A2EkFdaytOdnfnlBLCT1fGlBujmoGadp01TP/ItuP6lJTyopTyZQBLABTSkPcU4rUEAIgCWJ/jfjLJx//2b/PWi6iNRDyfMpusl26BIw7p9ZdsaxIxtYlJSvm2upJXngIAJjW3l+S4n0xy1113oa61NXFbXS9ifHjYwVKRF6Trk/J6DVPN6aTNuAzMz8JcLJ3eGYOEEOJuKeUEgAl1m5TybUtLlYEQogVACwDcu3y5k0Uxjd4X0+ov4eeff568XsTMDE739uLz69ctfV7yPj/2SY1Go/j1+fP4fGYmsS124waOnDkDIQRuz80ltvWfOwegOPI8ZVOT+IkQ4oiUckwIsQqAlFIWkuAkBmCx8ncAwKc57oeU8gCAAwDw4EMPze8l85jRaDRpUSG7voTL/+IvcObIEQDx9SLKFy4EANy6eRPR0VFP/+DJel6vMWil/ga15qQEUjrjZ2/fxuDYWFEEiWz6JM4DCCo1iou4cwLP1xHc6dMIIt4hDiFEIN1+PxscG5v35VS/hFZa8vWvJ5YmnVWuntZu2YJHGhvZL0FFRe83mEm6tef9JJuaRBDxq/tOZSjsSQBD+T6hlPKCEGK1EGI9gJhm5NLbAMJp9vuW0ZfNji9h5d13s1+CCuKFNByZ5PNbM1p73m+yCRJRKeVRAD8DACFEQ6FPqjQXpW4Lp9vvZ4GqKt0vqR1fQr+uY032cOMa2vkw+g0CgACQ2qZdVlqa6ND2u4xBQkp5VAixUplEtwrAfTaUq6jUh0Lz2kML/RJqV9364/Xr+G/33KO7+tayRx/15TrWZA+/JHrU+w0C8d/h7bm5eRMEVxdRSo+shsBKKS8p/18EcNHKAhUj7TKkZoxuUjvhIrW1aK2rw7kPPkDt/fej++TJpA5xNX3Bl7/wBd+tY032cFOix0JmWxv9BtXbqd7/+OO8n8trmLtMmygAAAAUCElEQVTJJcxMNjY4NoZIbS22rF0LAPifX/saACRup47K8NMoFbKXWxI9vnflSsE5m/R+g4eVJU1TFUunNcAg4UsVZWVoratL3FYnP1aUl6O1rg7D4+NOFY1y5PZOYb8nenSyv9AtGCR8aHp2Ft0nT2LL2rWoKC+HlBJCCEzfvImDp09z9S2P8EKnsJ8m1elNaLWiv9BrGCR8SP1iA0BrXR3+74cfovb++3Hw9OlEXwW5n1c6hf3QXGk0oTVSW4tIbW3RpuQAGCR8SdsJNzw+njS6KVJbW1RfcC9zU6ew36Wb0LqzoaGofzMMEj6VbUf4l+6/H+998AGAeD5+cg+3dAq7nRlrSDg5odXtuMY1cSUvlyqWdbELoY5qKvQ7bNQRXUwd1EZYk7CRE5leybv81CnsduygNsYgYROnMr26nduHeDrND53CXmD2hFY/YZCwSbqOMbd8Ed+7csXWfgkvDPGk4mHmhFY/YZ+ETdzeMaa26dq5kHy6IZ5E6bx35Yqt39VixpqETbwwc/NL99+P3ysjnezgpSGebBZzHw64sAeDhE3YMTafV4Z4slnMGmrgLauowOz0NAOvS7G5ySbhYBCR2tpEzSFQVeXaiW12VeP1hngCwPIHHrDl+bPFZrFk0dFRDOzejTeeew4Du3cjOjqa1zFG+vpwddEi/PDwYVxdtAgjfX1ZHYvNTPZiTcJGXugYU5uc7OjEDobD+OTSJXx49mzS9uj58/jiypWWXFXm02zkpWYxq5lVqxobHMTUihXo7O1FZ28vAKCtqSljyhE1QFjd1MTh6newJkHz2NnW+/H778/bVshVerqrXPUEp57c1RNcpqtXo+YvtzWL2cGsWlVZRQU6enqStnX09KBMWXM9HTsCRP+5c4k+RHW4+mg0aunzuhWDBDnKzKv0TEEg3xMcZz7fYdbnNTs9jfbm5qRt7c3NaZfNtauZKd1w9WLEIEGG7BhmaOZVeqYgkO8JLhgOozYSSZSpKhAo2tX7Cvm81Fre8b17cevmTSycmEBbUxPm5ubQ1tSEysuXMwZeO2q5bh+ubjf2SZAu9ceY75DYbEeumLloTaYgUMhoKs58jsv381JrebWRCOpaW3Gyuxtn+/pQ8Yc/YN+mTVg6PY1QY6Pue2xXP4TKC8PV7eRITUIIERFCrBdCtBns36P832JvyUhPrrUJvZEr7xw6hHNHj867r5lX6ZmuctlsVLh8P6+xwUHURiJYu2ULAGDtli14pLERZQsX4vHnnkPDzp1pj2FnP1l9KISy0tKkbcU8XN32moQQogYApJSnhBBBIUSNlPJCyt1ahBARAK12l4+S5TPayWjkyodnz+qOWjLrKj3TVS4T5pkjn8+rrKICda13fs7lyu3x4eG0j3NiuGsueZyKYRSUE81NTwE4qfwdBbAeQGqQeFJKecrWUpGhXGdiqyNX1AABxEeu7Nu0ydJV1bIJAmw2so+2yfH6H/+Ik93dWLtlC8orKnBzehqnDx7M2FGdbQ1CPVlXlJVhena24JN1NsPViyVppxNBIgBgUnN7ic59aoQQAFAjpexM3ak0Q7UAwL3Ll1tRRkqRy+JERiNXKv/wB8vKp2IQcAe1yXFqxQp09PSgvbkZ7xw6BACoa23F6YMHE30UenKpQagn60htLVrr6tB98qQtJ2svJO00gytHN0kpO5WaxBIhxHqd/QeklKullKsXLV7sQAmLU7ZJAEP19VgYjSaNXFmojDEvxrkFfpVuToq2ybGkpASdvb2YCQZx5s038Y/f+U6ij8KMjurBsTFEamuxZe1aAMCWtWsT61JbqVhGQVlSkzDocI4qJ/4YAPXMHgDwacpjIwAgpexX9vknJDvA7DbTbJqetDOp923aFK9BCMFOYh/JNPM6XZPj4889p3tM7cVHLh3VFWVlaK2ru3O7vBytdXUYHh/P5SXlrFhGQVkSJKSUB9LsPgJgtfJ3EMApABBCBKSUMcT7KdSpjfcB6LaijMXAyjZT9Qdt1PRUu3EjvrhyZWKOgpOdxMzgar50c1KC4bBhk+NSgz6IQoa5Ts/OovvkSWxZuxYV5eWYvnkTB0+fxnRK+cxWLEk7be+TkFJeEEKsVpqRYpqRTW8DCCv7W4QQkwA+0hn5RFmyqs002zkUbugfYAZXa2SakxKqr8dIXx/ampoSfRKVly8j1Ng4bw7Nskcfxf944om8y6KerAGgta4OB0+fTvRRWKlYVrNzZDKdXk1DShlOt59yZ0ebqd2r2eUq0xUv5SfTxETtSDPtZDkA8ybVnXnrLQTuvRdfeeSRvMqiPVkPj49jenbWtgzLXkjaWSjOuPaobPoa0rWZmtFXoZ1DAWQe9VSIfJuMmMHVGtnMvNarSQ7s3j1vUh0AXDh2LO8gARTHydoprhzdROllm6XSaOboA8uXm5bl8kv332/50qf5Zm8FmMHVKvnOvFYn1ZUr2V7VSXULddYVIXdgTcKDsu1rMGoztaKvwspaRaYmo3S1DDNzQ1GyfPqcPv/8c91JdTMWdzJT/hgkPCiXvga9avjhkRHDx+89fjzvGauFJgU0kq7JKFPHNFNxuEPi4qGxEed+/nMAmkl1R4/ioe99z8niURoMEh5U6PhsvcfLuTkEZ2bQ8/d/j+b2dvQpgSTfmoWZHdrpOkmz6Zh2wyirYqY3vPXCsWMYHx7GzOwsHvre9wrqjyBrsU/CgwrNUqn3+ODMDHo7O1FSUoLezk6smJrKe8aqto/CjH6KdNlb2THtXurnr+23AoCvPPIIvtXZiUdfegnf6uxkgHA5BgkPCgeDiNTWJmoOgaqqnIb8pT7+3kAAPR0dSffp6ehARQGdiWZ2aKfrJGXHtPtoLw7sTPFN1mBzk0cVOuRP+/jdAwNobm9P2t/c3o7ppUsLKiNgXoe2tslI7ageHx7GrZs3IUpKIOfmEvdlx7Rz1M/54wUL4llZf/tbU7KyknMYJAj1oRD6RkbQ1NaGno4ONLe343JlJRpNSi+g7dAutK9CL7toxaVLWFhZiZtTU7Z1TOcyb6MY0oJoa4sfL1jgSFZWsgaDBCUNld20bx+mly5FowVXfonmpyxTjusxWtDoT65dw1MvvmhaWdOJjo7ibF8f5pRhxDdiMZzt6wMwP9VHMaQFSW1aen1gYF5WVgC+S6FdLNgnQQDigWJnQwOee/xx7GxosPTHnNqxnUufhZpdVKujpwdlyuQsO5z/9a8TAUI1d/s2zv/61/Pum270ldcZ9T2oWVkrysvjt5WsrIX0cZFzWJMgR6R2aBa6oJFRdlErzHz+edbb/Tb6KjWgp36Oo9Eo/nj9On757rvY+Od/bmtWVrIGaxI+MRqNYvfAAJ574w3sHhjIK8WGk3JZ0Kjy8uWkBY0qL192bUe1X0ZfpdYaUoe1AnfSxXxnzRo8/eijGLt0CQASWVn9lkK7WLAm4QN2rrVr5cLvqX0WKm3twii7qJ3t++VKJ7ne9lReTguSqdaQKnWFuNDKlfjlO+/gzTNn8J01a9gf4VEMEj5g11q7dgUj7clIb0SU0zOoH/72t3HmyJGkYbeipAQPf/vb8+6bTVqQ1PUVnB79ZOYKcU9/85s4PjrKAOFhDBI+YNdau04s/G7GiCiz5ZoPKl1QU0c/addXcGL0U661Bj1OrRBH1mKQ8AG71tp1cuH31El5gLMBw6zazNjgoO76ClYviqTX91Po7GinVogjazFI+IBda+06vfC70YgolRtqGblS11dQqesrjA8Pm/5cZtQW0nFyhTiyDoOED9i11q5eMAKAB5YvN/V5sqXXd6HySsCYnZ7WXV9h1qQhvVYHhlRcIc5/HAsSQogaKeUFg30RADEANVLKTntL5k12/DjDwSAuffIJzn74YdL288pw2/c//tixBeHTBQyVGwOHOvoJ0KyvoPRR5MJo6DAT7FGhHAkSQoj1APYAmNfoKoSoAQAp5SkhRDBdMCH7vf/xx/O2zd6+nRQ4rByCmw2jE2Nq85TKKHic/d3vcOy3v0XZwoWYnZnBE3/6p3jkK18xrZxAcif4+PAwZqenMy4DyoBAdnIkSCgBYNJg91MATip/RwGsB8Ag4RLZdlJbPeopH3onUaOkg2d/9zv8/D/+Axs1Sep+rgQ+KwJFtp3UTMFNdnNjn0QAgDaALHGqIDSfUee1HjtGPRVKb9QUABx57z1s1ElSd+zCBdODhCqbHFYMDmQ3NwaJjIQQLQBaAOBehzpNi4l2lnVleTlKS0pwWzORzIhdo54KpXfirfrtb+dNDGutq8Pw+Lgpq+3lUhYiJ1kSJJSTeKqolPJUFg+PAVis/B0A8GnqHaSUBwAcAIAHH3pI5ltOyix1lvXUzZsoEQJ3LVyIz2dmEKiqwgPLl+N8NGr5EFw7pZsY5qcTuXoBUFFWxsWBSJclQUI5iedECBGQUsYAHAGwWtkcBJBNYCGL6M2ynpMS5QsW4H83Nia2rfziFy0fgmunYpgYNhqNom9kBCumphKLTfWNjADg4kB0h1OjmyIAVgshIlLKfmXz2wDCUsoLQojVygioGEc2OSvbWdZ+Gx9fDBPDBsfGsGJqCr2dnejtjI80b2prc92AA3KWU6Ob+gH0p2wLa/7OuSZC1nB6lrWT/Bb4UlWUlaGnoyMRIACgp6MDm/btA2Btxl/yDk92XJN97Er5YYQnKutMz86iub09aVtzezumly61Nf08uRsXHaK0wsEgIrW1iZpDoKrKtmYX9USl1mTUE5XZCyqpCzbtPX7ckws25as+FMLlyko0tbVhbm4OTW1tuFxZifpQKG3GXyourElQRk41u9iRmryYO2+1/S6b9u3D9NKlaFRqaoeV9yCVF+a+kLkYJMi17EhNXuydt0YXAMXcF0XJ2NxErmV0QjLzRKV23mr1dHSgoqzMtOdwg1zXQK8PhVBWWpq0zetzXyg/DBLkWnacqAw7b320mlo+fTtO9kWRu7C5iVzLjnUy6kMh9I2MoKmtLdEncbmyEo0+umLOt2/H70OAKTsMEuRqVp+o0nXe+oWTy86S9zFIUNHz+xUzO6GpEAwSRD6lnYiYKlPfDicxkopBgsiHUmdMa2U66XO2NWlxdBORD+l1VgPxALGzoSHtyZ6zrUmLQYLIhwrprGZHN2mxuYnIRG5pyy+ks5od3aTFIEGUIt8TvZva8gvJ3ut05l9yFwYJIo1CTvR2JCTMViETEe2YxEjewSBBpFHIid5tbfmFzP/w+9wRyh47rok0CjnR25GQkMhuDBJEGoWc6Jk5lfyIQYJIo5ATPTOnkh851ichhKiRUl4w2LdHSvmCEKJFSnnA7rJR8Sq005Zt+eQ3jgQJIcR6AHsAhA3u0iKEiABota9URHE80RPd4UiQkFKeEkJMprnLk1LKU7YViIiIdLl1CGyNEAIAaqSUnU4XhoioWLmy41pK2anUJJYoTVNEROQAS2oSQogWnc3RbJqQlL4ISCn7AXwKYF7jsHL8FgC4d/nywgpLRESGLAkS+YxIEkIEpJQxAFHlHwDcB6Db4PgHAODBhx6SBRSViIjScKS5SaktrFZrDYq3AUAZFtuo7PvIaJgsERFZz6nRTf0A+lO2hTV/c24EEZELuLLjmoiI3IFBgoiIDDFIEBGRIQYJIiIyxCBBRESGGCSIiMgQgwQRERlikCAiIkMMEkREZIhBgoiIDDFIEBGRIQYJIiIyxCBBRESGGCSIiMgQgwQRERlikCAiIkMMEkREZEhI6e0looUQVwD8l8PFWArgqsNlyAfLbS8vltuLZQZY7kz+u5RyWTZ39HyQcAMhxHkp5Wqny5ErltteXiy3F8sMsNxmYnMTEREZYpAgIiJDDBLmOOB0AfLEctvLi+X2YpkBlts07JMgIs8TQtRIKS8Y7IsAiAGokVJ22lsyYxnKvEdK+YIQokVK6WjgYE3CBEKImjT79ij/t9hXouxkKHdECLFeCNFmZ5n8INN759b3Notyu/K7LIRYD+BnBvtqAEBKeQpALN133k7pyqxoEUJ8BCBqU5EMMUgUyEsftpYXf1iA+09kmd47t763WZbLld9lpcyTBrufQrwWAcTLvd6WQmWQocwA8KSU8j7lfo5ikCiQlz5sLS/+sDxyIsv03rnyvUV25XLldzmDAJK/50ucKkiOatxS22SQsJ5rPuwcuPWH5YUTWab3zq3vbTbl8uJ32ZOklJ3Kd3iJUut3DIOExdz0YfsAT2QO8uh3OQZgsfJ3AMCnDpYlK0qTakS5+SmAoJPlWeDkk3uBQdt2NJsrVfWDllL2w+YPu5Byw4M/LJU6ekUIUSeEWO9AjSLTe+fW9zZtuZz8LudDCBGQUsYAHAGgzmAOAnBtU5mmzFHcaS69D0C3c6VikMgon+FnbviwCyy3Yz+sDMHNCycy3ffODe9tBpnK7aoTl5byua8WQkSUzx4A3gYQllJeEEKsVmo+MaMhp3bLoswtQohJAB85XWbOkyiQ8mH/DMBfqx+2EGJUShlW/m5BvIkk6LIx2tmUO4p4uV0xwUfpqF4tpTygNCedUn5QASml2pEdVf7uBtDtxA9M771z+3sLZF1u132XyVoMEuQpPJER2YtBgoiIDHF0ExERGWKQICIiQwwSRERkiEGCiIgMMUgQEZEhTqYjMoky9ySI+BDdhwH8gzIRjcizWJMgMoEQIqhMSlSDwhE1QAghajyU64goCYMEkQmklGrKijCUmeCa3U8hnkaEyHMYJIhMoFnbIqhJEaI64kSZiMzAPgkic6wXQgQBnFSaltItREXkGUzLQWQxJZ/UfWBHNnkQgwQRERlinwQRERlikCAiIkMMEkREZIhBgoiIDDFIEBGRIQYJIiIyxCBBRESGGCSIiMjQ/wfejODiFyP7twAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1cfae0e3b38>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "plot_contour_2d_classification(circle_X, circle_y, circle_svm_gaussian_classifier,\n",
    "                               delta, r\"SVM classifier with gaussian kernel for $\\sigma = 1$\",\n",
    "                               lambda: add_important_vectors(circle_svm_gaussian))\n",
    "print(\"Accuracy: \" + str(accuracy(circle_y, circle_svm_gaussian_classifier(circle_X))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It comes as no surprise that arguably the most popular kernel is able to separate the points (almost) perfectly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2.8\n",
    "In this final task, we want to take a glimpse at a real world application of the support vector classifier. To that extent, we want to classify 28x28 pixel images of handwritten digits using the *Scikit-Learn* python library. We use the *MNIST* data set that is provided by Yann LeCun.\n",
    "\n",
    "Let us acquire it first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download(filename , source=\"http://yann.lecun.com/exdb/mnist/\"):\n",
    "    \"\"\"Downloads file\n",
    "    :param filename: name of the file to download\n",
    "    :param source: url to download the file from\n",
    "    \"\"\"\n",
    "    print(\"Downloading %s\" % filename)\n",
    "    urlretrieve(source + filename , filename)\n",
    "\n",
    "def load_mnist_images(filename):\n",
    "    \"\"\"Returns MNIST images for the specified file. May download the file if not locally available\n",
    "    :param filename: name of the image file.\n",
    "    \n",
    "    :returns: ndarray. 3 dimensional matrix containing the images.\n",
    "    \"\"\"\n",
    "    if not os.path.exists(filename):\n",
    "        download(filename) \n",
    "    with gzip.open(filename, \"rb\") as f:\n",
    "        data = np.frombuffer(f.read(), np.uint8, offset=16)\n",
    "    data = data.reshape(-1, 28, 28)\n",
    "    return data / np.float32(256)\n",
    "\n",
    "def load_mnist_labels(filename):\n",
    "    \"\"\"Returns labels for MNIST images for the specified file. May download the file if not locally available\n",
    "    :param filename: name of the labels file.\n",
    "    \n",
    "    :returns: ndarray. contains the labels.\n",
    "    \"\"\"\n",
    "    if not os.path.exists(filename):\n",
    "        download(filename)\n",
    "    with gzip.open(filename, \"rb\") as f:\n",
    "        data = np.frombuffer(f.read(), np.uint8, offset=8)\n",
    "    return data\n",
    "\n",
    "X_train = load_mnist_images(\"train-images-idx3-ubyte.gz\")\n",
    "y_train = load_mnist_labels(\"train-labels-idx1-ubyte.gz\")\n",
    "X_test = load_mnist_images(\"t10k-images-idx3-ubyte.gz\")\n",
    "y_test = load_mnist_labels(\"t10k-labels-idx1-ubyte.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the algorithms from the *Scikit-learn* library, in particular the support vector classifier, expect the data to be a matrix with only two axis, we first have to unroll the axis 2 and 3 into a single one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_unrolled = X_train.reshape((len(X_train), np.prod(np.shape(X_train)[1:])))\n",
    "X_test_unrolled = X_test.reshape((len(X_test), np.prod(np.shape(X_test)[1:])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having a the unrolled data, we can now use *GridSearchCV* from the *Scikit-learn* to perform a 5-fold crossvalidation to train support vector classifiers for a range of different parameters for the regularization parameter *C* and *gamma*, which is relevant for the **RBF** kernel. We train the classifiers on a random subset of 500 examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best mean accuracy score: 0.878\n"
     ]
    }
   ],
   "source": [
    "cv_k = 5\n",
    "subset_size = 500\n",
    "\n",
    "picked_ids = np.random.choice(range(len(X_train_unrolled)), subset_size, replace = False)\n",
    "\n",
    "X_train_sub = X_train_unrolled[picked_ids]\n",
    "y_train_sub = y_train[picked_ids]\n",
    "parameters = { \"C\": [1, 10, 100], \"gamma\": [1e-1, 1e-2, 1e-3] }\n",
    "\n",
    "svc = SVC(kernel = \"rbf\")\n",
    "clf = GridSearchCV(svc, parameters, cv=cv_k)\n",
    "\n",
    "clf.fit(X_train_sub, y_train_sub)\n",
    "\n",
    "best_params = clf.best_params_\n",
    "best_score = clf.best_score_\n",
    "\n",
    "print(\"Best mean accuracy score: \" + str(best_score)) # this is valid as the default scorer for SVC is the mean accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having obtained the best performing parameter combination we train a support vector classifier with these parameters on the random subset of 2000 examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "subset_size_final = 2000\n",
    "picked_ids_final = np.random.choice(range(len(X_train_unrolled)), subset_size_final, replace = False)\n",
    "\n",
    "X_train_final = X_train_unrolled[picked_ids_final]\n",
    "y_train_final = y_train[picked_ids_final]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=10, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma=0.01, kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_classifier_final = SVC(kernel = \"rbf\", C = best_params[\"C\"], gamma = best_params[\"gamma\"])\n",
    "svm_classifier_final.fit(X_train_final, y_train_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let us evaluate our classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 965    1    1    1    0    7    3    1    1    0]\n",
      " [   0 1121    3    2    0    2    3    2    2    0]\n",
      " [   8    3  973   10    4    2   11   10    8    3]\n",
      " [   6    2   20  928    1   19    2   10   14    8]\n",
      " [   1    2   10    0  926    0    1    4    3   35]\n",
      " [  11    4    2   16   11  819   10    5    7    7]\n",
      " [  15    4    4    0    8   17  902    0    8    0]\n",
      " [   1   10   23    4    4    0    0  973    2   11]\n",
      " [   7    5   11   12   10   12   11   14  883    9]\n",
      " [   8    6    1    9   29    6    1   23    7  919]]\n",
      "Accuracy: 0.9409\n"
     ]
    }
   ],
   "source": [
    "predictions = svm_classifier_final.predict(X_test_unrolled)\n",
    "print(confusion_matrix(y_test, predictions))\n",
    "print(\"Accuracy: \" + str(accuracy_score(y_test, predictions)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, our classifier obtains a very good accuracy far above 90% and most of the times reaches a higher accuracy score than during cross validation, which is probably due to the size of the sets the classifiers were trained on.\n",
    "\n",
    "### On the use of different sets\n",
    "Using the technique of crossvalidation, we try to minimize the influence the specific choice of training data has on the evaluation of the parameters, to receive a parameter tuple which is most likely to generalize well. However, this approach is of course not completely safe. It is dependend of the size of the set we train on and the random training examples we use. \n",
    "\n",
    "If our chosen set is to small, it is more likely to receive a skewed training set (which means the training set may only contain very few examples for one of the classes). Then our training set is not representative and we should not trust the results obtained by the crossvalidation. This of coure may also happen if the chosen set has a decent size (as compared to size of the set one intends to train with the optimal parameters), but it is less likely. This is the main factor: both sets have to accurately represent the data.\n",
    "\n",
    "Furthermore, we should consider that different combinations of parameters may scale better with an increasing number of training examples (especially with respect to the regularization parameter). Because of that, we should not perform crossvalidation on a training set that is far smaller than the training set we intend to train on with the best parameter tuple.\n",
    "\n",
    "Keeping these factors in check it is a valid approach. Our model is supposed to generalize, so the choice of parameters should be independent of the training set, **if** the training set accurately represents the domain and range of the unknown function the model is supposed to learn."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}