project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DSC 80: Project 03\n",
    "### Due Date: Thursday, November 10, 11:59:59 PM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* ⚠️ **There is no checkpoint for this project** ⚠️ However, we suggest that you try to self-impose the checkpoint in order to make sure that you are able to complete the project on time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instructions\n",
    "\n",
    "Welcome to Project 03! This project is very different from the previous projects in that it is not autograded. Instead, you'll choose one of three datasets to analyze. Rather than turning in a `.py` file, you'll submit a Jupyter notebook containing your analysis. The three datasets you can analyze are described by the other notebooks in this directory:\n",
    "\n",
    "- `league.ipynb`: *League of Legends* competition data\n",
    "- `stocks.ipynb`: data on stock trades made by American politicians\n",
    "- `outages.ipynb`: data on power outages\n",
    "\n",
    "This project covers *no new topics*; it is an opportunity for you to use what you've learned in DSC 80 so far to carry out an open-ended data science project. As such, this project is an **excellent thing to put in a portfolio** and show to potential employers.\n",
    "\n",
    "No matter which data set you choose, your project will involve the following steps:\n",
    "\n",
    "1. Pick **one** of the three datasets described in the notebooks in this directory.\n",
    "\n",
    "1. Narrow down a few questions and lines of inquiry to pursue in the dataset of choice.\n",
    "1. Assess the quality of these datasets via exploratory data analysis, placing your results in the context of how the data were generated.\n",
    "1. Assess the mechanism of missingness for some relevant portion of the dataset.\n",
    "1. Ask/answer a question about the dataset using a hypothesis test or permutation test, being sure to discuss uncertainty of your result and possible shortcomings of your approach.\n",
    "1. Conclude with how you might improve your work and what data you might use to do so.\n",
    "\n",
    "## How to Organize and Submit Your Work for Grading\n",
    "\n",
    "### Summary of Findings\n",
    "Each of these steps will be summarized in the *Summary  of Findings* section at the top of the project notebook. This should include:\n",
    "\n",
    "* Introduce the dataset as you understand it and how it relates to the question you are investigating.\n",
    "* Describe data-cleaning steps and how it affects your analyses. Steps should be explained/justified in reference to the data generating process.\n",
    "* Describe the setup/results of your assessment on missingness, including how to interpret those results, your statistical confidence, and how they might affect your ability to answer questions about the dataset. \n",
    "* Describe the setup/results of your Hypothesis test, including being clear about your null/alternative hypothesis, your test-statistic, the significance level you used, and what conclusions you can draw from the results.\n",
    "\n",
    "Each of the above should contain *specifics* about your data-analyses (e.g. concrete numbers). The work/code/details for how you obtained your results should be included in your notebook, **below** the *Summary of Findings* section. You should clearly label what you are doing in your work using the 'markdown cells' in your notebook.\n",
    "\n",
    "### Style\n",
    "\n",
    "When doing the work that informs your summary, **you should write organized, readable code, broken into clearly delineated sections.**\n",
    "\n",
    "* Your work for each of the project sections described below should be completed in code cells underneath the markdown-headers of that sections name. \n",
    "* The output of each code cell (e.g. tables and plots) should have an explanation of is contained in them and why. This explanation should be:\n",
    "    - Written in a markdown cell above the code when you are explaining what the output of a cell is doing and how it relates to your question.\n",
    "    - Written as a code comment when explaining what the code is doing.\n",
    "    - Written in the title of a plot.\n",
    "* You should **only include work that is relevant** to posing, explaining, and answering the question(s) you stated in the summary. You should include data quality, cleaning, and missingness assessments, though these should broadly be relevant to the question at hand.\n",
    "* Do **not** include long dataframe output; if you want to include specific data from a dataframe in the writeup, display only the `head`.\n",
    "\n",
    "### Requirements: Cleaning and EDA (Exploratory Data Analysis)\n",
    "\n",
    "* **Clean the data** appropriately for your question (e.g. replace data that should be missing with `NaN`, create new columns out of given ones -- e.g. compute distances, scale data, get time information from time stamps).\n",
    "* **Univariate Analysis:** look at the statistics of relevant columns separately (e.g. their distribution and statistics) using tables and appropriate plots.\n",
    "* **Bivariate Analysis:** look at the statistics of pairs of columns to identify possible associations. Use scatterplots, plot conditional distributions, box-plots, etc. Also, you should examine and plot pivot tables. This will best inform interesting hypothesis tests!\n",
    "* **Interesting Aggregates:** Choose columns to group-by and examine aggregate statistics.\n",
    "\n",
    "### Requirements: Assessment of Missingness\n",
    "\n",
    "* Recall that whether data is NMAR or not is an assumption of whether the missingness is explainable by observed data or not. In this section, you will investigate the likely missingness mechanisms of your data as being missing at random. Before making this assumption, address in the summary whether you believe the data is NMAR -- explain your reasoning and any additional data you might want to obtain that could explain the missingness (thereby making it MAR).\n",
    "* Pick a column with non-trivial missingness to analyze.\n",
    "* Perform permutation tests to analyze the dependency of this missingness on other columns.\n",
    "    - Find at least one column for which missingness is dependent and one for which missingness is not dependent.\n",
    "* Interpret the meaning of the permutation test results with respect to your data and question.\n",
    "\n",
    "### Requirements: Hypothesis Test\n",
    "\n",
    "* Formulate a hypothesis and perform a hypothesis test. You can use the \"sample lines of inquiry\" in each project notebook for inspiration or you can create your own.\n",
    "* Be sure to explicitly state the (null and alternative) hypothesis, the test-statistic, the significance level, the resulting p-value and results. Justify why these choices are good choices for answering the question you are trying to answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submitting Your Project\n",
    "\n",
    "You will submit the notebook with your write-up (and code) to Gradescope; you will **not** be turning in a `.py` file.\n",
    "\n",
    "1. Save your notebook (e.g. `league.ipynb`) as a PDF file. To do this, use your web browser to print the page (or open up print preview from File -> Print Preview); choose the print to PDF option (this is more reliable).\n",
    "2. Upload the file to the Gradescope assignment for the dataset you chose. For example, if you did `league`, then upload your pdf to the assignment `Project 03 - League`.\n",
    "3. You are done! (You do *not* need to upload this notebook)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}