API_Documentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UMedia API Documentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "This tutorial introduces Python code you can use to access data and metadata from UMedia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Installation of Python3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will illustrate how to use Python to obtain data from [UMedia](https://umedia.lib.umn.edu/). We will use Python3 (more specifically, version 3.7.4), which you can download from the [Python website](https://www.python.org/downloads/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Required packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you're ready to use Python, let's import several required packages: \n",
    "\n",
    "* 'json' is a python built-in module that functions as a json encoder and decoder. \n",
    "* 'requests' is a Python module that you can use to send all kinds of HTTP requests. In this tutorial, you will learn how to use requests to send simple HTTP requests in Python. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Formatting URLs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating URLs is an important step to use the UMedia API. We need the URLs to make the request to the website. If the API is the car, the URL is the steering wheel of the car. Without it, the API won't go in the right direction.\n",
    "\n",
    "For more info on how to format URLs for UMedia, see the Libraries' [Digital Collections APIs page](https://www.lib.umn.edu/digital/apis)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Using the Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Download JSON from UMedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the ```requests``` and ```json``` libraries, and their built-in functions, to download data from UMedia. We'll focus below on the \"Digitizing Immigrant Letters\" collection. \n",
    "\n",
    "Let's create a URL and store it in a variable called ```my_url```. We will concatenate strings using the \"plus operation\" in Python to combine different elements of the URL: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_url = \"https://umedia.lib.umn.edu/search.json?\"\n",
    "my_filter = \"facets%5Bcollection_name_s\"\n",
    "my_key = \"%5D%5B%5D=Digitizing+Immigrant+Letters\"\n",
    "\n",
    "my_url = base_url + my_filter + my_key\n",
    "print(my_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look at the results for the URL above, you see they only include the first 20 items from the search results. To access data for up to 50 items at a time through a single API call, we can add ```&rows=50``` at the end of the URL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_item_num = \"&rows=50\"\n",
    "\n",
    "my_url_50 = my_url + my_item_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep things simple, we will use the ```my_url``` variable here, which includes 20 items. \n",
    "\n",
    "Now let's use the ```get()``` function in the ```requests``` package to access the data from our ```my_url``` variable: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(my_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variable called ```r``` includes our search results as a response object, which can be difficult to read and understand. You can view the contents of the object by running ```print(r.contents)```. \n",
    "\n",
    "We can use the ```loads()``` function in ```json``` to read the result into JSON, which will be easier to work with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = json.loads(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The information has been stored in the variable called ```data```, which is a Python list of Python dictionaries. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at how many items are stored in the ```data``` variable. This can be done with the Python ```len()``` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access an item in the data list by referring to its index: ```data[0]``` refers to the first item in the list, ```data[1]``` the second, and so on. We use a zero instead of a one for the first item in the list because Python (and many other programming languages) start indexes for lists from zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to note that because our ```data``` list starts from the index ```data[0]``` the index will end at ```data[19]```. For a list of 20 items we can't look for ```data[20]``` because it is actually indicating the 21st item in the list (which doesn't exist)! Looking for ```data[20]``` will give us a \"list index\" error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python is comfortable with this\n",
    "data[19]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python would complain for this\n",
    "data[20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next concern would be: what specific data elements are stored in each item from the ```data``` list? \n",
    "\n",
    "To answer this question, we need to dive into the inner dictionary and explore what information is stored there. Each dictionary includes a list of keys and values. We can tell it's a Python dictionary instead of a Python list becuase it's contained by curly brackets: \n",
    "```\n",
    "{'id': 'p16022coll264:133', 'collection_name': 'Digitizing Immigrant Letters'}\n",
    "```\n",
    "In the example above, the first key is 'id' and its value is 'p16022coll264:133'. The second key is 'collection_name' with a value of 'Digitizing Immigrant Letters'.\n",
    "\n",
    "We can use the following code to store all of the keys for the dictionary in ```data[0]``` into a variable called ```keys``` and print them out using for-loop. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = list(data[0].keys())\n",
    "\n",
    "for i in range(len(keys)):\n",
    "    print(keys[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above results, we can see all of the keys that we can use to obtain values such as title, creator, transcript, page count, type, date added, etc. We can reference these keys to take a look at the title and the transcript of the first item: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[0][\"title\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data[0][\"transcription\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To print out the titles for all 20 items we grabbed from UMedia in the ```data``` list, we can use another for-loop: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(data)):\n",
    "    print(data[i][\"title\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can work with our ```data``` variable to obtain more information about specific items in our UMedia search results. Let's consider how to save the transcripts for all of the items in our Python list to a text file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function takes two arguments: the first specifies the name of the file and the second specifies the task we want to do. In our case, we want to write data to the file, so  we'll use the symbol \"w+\" in the second argument. See [the Python documentation](https://docs.python.org/3/library/functions.html) for more information about the \"open()\" function and its usage. \n",
    "\n",
    "The following line of code does three things: \n",
    "\n",
    "1. Creates a file object called \"f\"\n",
    "2. Creates a text file called \"letters.txt\" on your computer\n",
    "3. Specifies that we want to write to the text file, ```\"w+\"```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open(\"letters.txt\", \"w+\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After opening (or creating) the file, we can write data to it using the ```write()``` function. We can use a for-loop to iteratively write out both the transcripts and titles for each item into our letters.txt file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(data)):\n",
    "    f.write(data[i][\"title\"] + \"\\n\" + data[i][\"transcription\"] + \"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There should now be a file called \"letters.txt\" in the same directory with this code. When opening it, you will see the title along with the transcript of each item. \n",
    "\n",
    "Remember to close a file every time you open one. This avoid memory leak issues, which can lower your CPU performance. The code to close your f file object is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Dowload Objects from UMedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will discuss how to use the API in Python to download digital objects from UMedia. We will focus here on how to download image files. \n",
    "\n",
    "To download images, we can use another built-in module in Python called ```shutil```. This module allows us to perform a variety of operations, such as copying and deletion, on local files. See [the Python documentation](https://docs.python.org/3/library/shutil.html) to learn more about ```shutil```. \n",
    "\n",
    "\n",
    "First import the tool:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a URL for the image we want to download. As an example, we will use the image from [How to construct IIIF calls to UMedia documentation](https://www.lib.umn.edu/digital/apis#iiif). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_url = \"https://cdm16022.contentdm.oclc.org/digital/iiif/p16022coll208/4833/full/full/0/default.jpg\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can use the ```requests``` package to open the url image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = requests.get(image_url, stream = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to set stream to 'True' in ```get()```, which we didn't do previously when working with UMedia metadata. This is because the image information is binary and not available in the JSON format, as before. \n",
    "\n",
    "Next, we can open the file to store the image that we called above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_file = open('local_image.jpg', 'wb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second argument of the ```open()``` function is ```wb``` instead of ```w+``` because we need to write binary data to the file, indicated by the 'b' in 'wb'. \n",
    "\n",
    "Before writing the data to a 'local_image.jpg' file that we just created, we first need to set the ```resp``` variable object to ```raw.decode_content = True```! Otherwise the size of this file will always be zero, and thus cannot be opened by any computer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp.raw.decode_content = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can write the image data to the file. This is done by one line of code, using the ```copyfileobj()``` function from the ```shutil``` module. Don't forget to delete the image url response object before celebrating: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write the data into the local_file\n",
    "shutil.copyfileobj(resp.raw, local_file)\n",
    "\n",
    "# remove the image response object\n",
    "del resp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the code above, you should find that the image called \"local_image.jpg\" is in the same directory as the code! Now you know how to download images from UMedia. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the end of the tutorial. Thank you for reading! "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}