portaltasks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Portal Tasks and the ArcGIS API for Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Portal for ArcGIS is a big interface for and facilitator of GIS users. It provides an interface for users to interact with a bunch of server components that get stitched together in one big federation of glory. The default and primary way we interact with it is the user interface--like one big website or application. I've noticed a lot of organizations with Portal for ArcGIS haven't yet explored the alternative routes to carry out tasks.\n",
    "\n",
    "This post is a short demonstration of one of those alternative routes Esri provides: the ArcGIS API for Python. The API has grown to encompass a whole lot more than I can cover in one post, or even a lot of posts, so this will be a quick but useful working demo for interrogating some components of your Portal. Also covered is a simple method to upload PDFs from a folder on disk to your Portal, share those PDFs, and build links to them for your organization's users to consume.\n",
    "\n",
    "All of the demo sections will work equally well between Portal and ArcGIS online except for the PDF access. ArcGIS Online hosts this kind of content through Amazon and therefore building a direct link and gaining access is different and not covered here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We start by importing our python packages. All of these packages should come with your python installation\n",
    "# assuming you have at least python 3.5. If you have ArcGIS pro installed you will already have a sufficient\n",
    "# version of python. The API provides good access so long as you have at least Portal 10.5. I haven't tested\n",
    "# compatibility but I would suspect these simple operations to work if you have this version or higher.\n",
    "\n",
    "import os\n",
    "import time\n",
    "import arcgis\n",
    "from arcgis.gis import GIS\n",
    "\n",
    "# IPython is not required to interact with Portal, it is included here to use the display class while in Jupyter\n",
    "# Notebooks which is a web application that lets you run interactive python code.\n",
    "\n",
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The username and password variables are the values you use to log into your Portal (or AGOL).\n",
    "\n",
    "username = 'YOURUSERNAME'\n",
    "password = 'YOURPASSWORD'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For Portal access, the URL will either need to be the Web Adaptor URL used to access the environment through web tier\n",
    "# authentication such that users have access to Portal from the public web OR you will need to be on your network (or\n",
    "# connected through a VPN) if your Portal is not accessible from the public web. Most of the time the URL will be the URL\n",
    "# you use to get to the main Portal site. If working with AGOL, use the AGOL URL like https://COMPANY.maps.arcgis.com\n",
    "\n",
    "# This object provides helper objects to do all the things we want, so we build this first to get access to the organization.\n",
    "\n",
    "gis = GIS(r'YOURPORTALURL', username, password, verify_cert=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we run a search and get a list of all users using a wildcard.\n",
    "\n",
    "users = gis.users.search('*')\n",
    "\n",
    "# This statement prints the number of users in the list, ours is up to 100 now!\n",
    "\n",
    "print(len(users))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This statement prints the users one-by-one.\n",
    "\n",
    "for u in users:\n",
    "    print(u)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This variable will serve as the name of a group we will create (all without a UI!).\n",
    "\n",
    "groupname = 'DELETEME'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The API is nicely organized, so \"gis.groups.create\" does what you might expect. We just need to supply it with some\n",
    "# properties like a title, tags, etc. We can also set the access level right at this point of creation.\n",
    "\n",
    "print('creating test group')\n",
    "temp_group = gis.groups.create(title=groupname,\n",
    "                              tags='delete',\n",
    "                              description='Temporary group',\n",
    "                              snippet='placeholder description',\n",
    "                              access='org',\n",
    "                              is_invitation_only='False')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now that we have created a group, lets search for and create a list of groups with a query. We can also go the the Portal\n",
    "# Group section to find the group in the list.\n",
    "\n",
    "group_search = gis.groups.search(\"title:{} AND owner:{}\".format(groupname, username), outside_org = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will print out the groups found in our search to make sure we have just our new group selected.\n",
    "\n",
    "for group in group_search:\n",
    "    print('group name: ' + str(group.title))\n",
    "    print('group access: ' + str(group.access))\n",
    "    print('group members: ' + str(group.get_members()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# There was only one group in our list, so lets select that item and use it to send a delete command.\n",
    "\n",
    "group = group_search[0]\n",
    "\n",
    "print('deleting group: ' + str(group.title))\n",
    "group.delete()\n",
    "\n",
    "# And that is all it takes, the group is gone!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Its not uncommon to get a request to use Portal as a nice organizational storage and access point for things like PDFs\n",
    "# that commonly accompany our GIS data. Let's start with a folder of PDFs we want to host on Portal.\n",
    "\n",
    "pdf_dir = r'C:\\pdf'\n",
    "pdfs = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the walk function in our os package lets us build a list of all the files in the pdf folder, which all happen to\n",
    "# be the PDFs we want to upload. We can confirm by printing out that list.\n",
    "\n",
    "for root, dirs, files in os.walk(pdf_dir):\n",
    "    for filename in files:\n",
    "        pdfs.append(os.path.join(root, filename))\n",
    "\n",
    "print(pdfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we will create another helper object called ContentManager and we will name it manager.\n",
    "\n",
    "manager = arcgis.gis.ContentManager(gis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since we have a list of PDF files with a path to their location on disk, we can go through that list one at a time to add\n",
    "# each file to Portal.\n",
    "\n",
    "for pdf in pdfs:\n",
    "    pdf_name = os.path.splitext(os.path.basename(pdf))[0]\n",
    "    pdf_properties = {'type':'PDF', 'description':'PDF upload for map viewer','title':pdf_name, 'tags':'pdf', 'access':'org'}\n",
    "    manager.add(pdf_properties, data=pdf)\n",
    "    print('adding PDF: ' + pdf_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Those PDFs should be in Portal. On AGOL the PDFs should show up in Content. Depending on the version of Portal they may \n",
    "# or may not be easily found in the user interface of Portal. But they are there! Lets run a search to find confirm our\n",
    "# PDF's made it.\n",
    "\n",
    "pdf_search = manager.search(query=\"owner:{}\".format(username), item_type='PDF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we can print the list and see that yes, they are on Portal.\n",
    "\n",
    "for pdf in pdf_search:\n",
    "    print(pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# But how can user's access these PDFs? Lets change how they are shared and build a URL link that we can drop in a field\n",
    "# of some feature in our GIS, or add to a document, or anything else we want to do. Remember that these will be accessible\n",
    "# according to how you share them with the function share below and so long as the users have access to Portal, such as\n",
    "# being on the same network or having access to Portal regularly. There are some authentication schemes that may prohibit\n",
    "# this kind of direct access. If all goes to plan here a list of click-able links will get spit out for you to test.\n",
    "\n",
    "portal_url_base = r'https://YOURDOMAIN/portal/sharing/rest/content/items/'\n",
    "\n",
    "for pdf in pdf_search:\n",
    "    pdf.share(everyone=True)\n",
    "    pdf_item = gis.content.get(pdf.id)\n",
    "    pdf_url = portal_url_base + pdf.id + r'/data'\n",
    "    print(pdf_url)\n",
    "    time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To clean these up for our demonstration, we will get a list of what we've just added. Confirm this list only contains\n",
    "# the items we want to now get rid of.\n",
    "\n",
    "pdf_search = manager.search(query=\"owner:{}\".format(username), item_type='PDF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally, we run a delete operation to nuke the PDFs.\n",
    "\n",
    "for pdf in pdf_search:\n",
    "    print('delete: ' + str(pdf.id))\n",
    "    pdf.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I hope this quick demo of some useful ArcGIS API for Python functions gives you an idea of how you can access your Portal in more programmatic ways. You can visit find a whole lot of samples at the github page here https://github.com/Esri/arcgis-python-api/releases/tag/v1.8.2 and you can dig into the depth of operations available in the other Esri products enabled through AGOL or the base deployment including Portal at the API reference page here https://developers.arcgis.com/python/api-reference/arcgis.gis.toc.html.\n",
    "\n",
    "As you can imagine these tools can make a big impact on adminstration of your overall GIS. It can also be used by regular GIS users to perform analysis and conduct data operations needed for various tasks and projects. Many of these use cases are sampled at the previously linked github page. You can also schedule scripts built with the ArcGIS API for Python using Esri's ArcGIS Notebook Server. For more info you can read up on it here https://enterprise.arcgis.com/en/notebook/latest/install/windows/introduction-to-arcgis-notebook-server.htm. \n",
    "\n",
    "I think ArcGIS Notebook Server is a topic and product that will get heavy use as time goes on and as organizations adopt it, so I will certainly write about it in the future. For now, think about how this alternative approach to conducting tasks in Portal might streamline your workflow or help you automate a routine task and free up some more time for the important things in life like studying UFOs or estimating the percentage of feldspar in various types of sand."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}