010-overview.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "rc(\"image\", cmap=\"gray\", interpolation=\"bicubic\")\n",
    "figsize(10, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deep Learning for Document Analysis, Text Recognition, and Language Modeling\n",
    "## Thomas M. Breuel\n",
    "## NVIDIA Research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Personal History\n",
    "\n",
    "- document analysis at IBM, Xerox\n",
    "- professor, AI/DL research group at UniKL\n",
    "- DL researcher at Google, NVIDIA\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# OCR History\n",
    "\n",
    "- neural network-based OCR and handwriting recognition\n",
    "- HMM-based OCR\n",
    "- LSTM-based OCR\n",
    "- LSTM-based layout analysis\n",
    "- LSTM-based language modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Objectives for Today\n",
    "\n",
    "\n",
    "- introduction to the most important architectures in DL related to document analysis and recognition\n",
    "- an introduction to PyTorch and some useful add-on tools\n",
    "- applications to document analysis and recognition tasks\n",
    "\n",
    "Maybe:\n",
    "\n",
    "- understanding large volume preprocessing, extraction, and loading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PROBLEMS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classic OCR\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td><img src=\"figs/book-page.png\" style=\"width: 4in\" /></td>\n",
    "        <td>$\\rightarrow$</td>\n",
    "        <td><img src=\"figs/book-recognized.png\" style=\"width: 5in\" /></td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classic OCR Pipeline\n",
    "\n",
    "<img src=\"figs/ocr-book-pipeline.png\" style=\"width: 10in\" />\n",
    "\n",
    "Followed by: reading order, post-correction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Word-Based, Grayscale OCR Pipeline\n",
    "\n",
    "<img src=\"figs/ocr-word-pipeline.png\" style=\"width: 10in\" />\n",
    "\n",
    "Followed by: reading order, post-correction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scene Text Pipeline\n",
    "\n",
    "<img src=\"figs/ocr-scene-text-rcnn.png\" style=\"width: 10in\" />\n",
    "\n",
    "(Jaderberg et al. 2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Human Reading\n",
    "\n",
    "<img src=\"figs/fixation.png\" style=\"height: 2in\" />\n",
    "\n",
    "<img src=\"figs/reading-saccades.jpg\" style=\"height: 4in\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Largely Solved Problems\n",
    "\n",
    "- OCR of flatbed scanned, printed texts achieves error rates of < 0.3% on Western scripts\n",
    "- layout analysis of simple text documents, books, journal articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Largely Open Problems\n",
    "\n",
    "- analysis and understanding of complex layouts\n",
    "- reliable reading order of complex / multi-column documents\n",
    "- table understanding\n",
    "- figure understanding\n",
    "- historical documents\n",
    "- handwritten documents\n",
    "- high quality reconstruction of camera captured documents\n",
    "- high quality OCR from camera captured documents\n",
    "- human-like document understanding\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Outline for Today\n",
    "\n",
    "- Introduction (this)\n",
    "- convolutional neural networks and classical theory\n",
    "- new era in deep learning (2012-now)\n",
    "- PyTorch and add-ons\n",
    "- OCR-specific deep learning techniques\n",
    "- sequence recognition\n",
    "- semantic segmentation architectures and layout analysis\n",
    "- sequence transduction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Code\n",
    "\n",
    "- most of the models I'll be talking about will be available after the conference"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "jupytext": {
   "formats": "ipynb"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}