pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas\n",
    "\n",
    "Pandas is a data analysis library focused on the cleaning and exploration of\n",
    "tabular data.\n",
    "\n",
    "Some useful links are:\n",
    "- [main website](https://pandas.pydata.org)\n",
    "- [documentation](http://pandas.pydata.org/pandas-docs/stable/)<sup>1</sup>\n",
    "- [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)<sup>1</sup> by\n",
    "  Jake van der Plas\n",
    "\n",
    "<sup>1</sup> This tutorial borrows heavily from the pandas documentation and\n",
    "the Python Data Science Handbook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import pandas as pd  # pd is the usual abbreviation for pandas\n",
    "import matplotlib.pyplot as plt # matplotlib for plotting\n",
    "import seaborn as sns  # seaborn is the main plotting library for Pandas\n",
    "import statsmodels.api as sm  # statsmodels fits linear models to pandas data\n",
    "import statsmodels.formula.api as smf\n",
    "from IPython.display import Image\n",
    "sns.set()  # use the prettier seaborn plotting settings rather than the default matplotlib one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We will mostly be using `seaborn` instead of `matplotlib` for\n",
    "> visualisation. But `seaborn` is actually an extension to `matplotlib`, so we\n",
    "> are still using the latter under the hood.\n",
    "\n",
    "## Loading in data\n",
    "\n",
    "Pandas supports a wide range of I/O tools to load from text files, binary files,\n",
    "and SQL databases. You can find a table with all formats\n",
    "[here](http://pandas.pydata.org/pandas-docs/stable/io.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv')\n",
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This loads the data into a\n",
    "[`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)\n",
    "object, which is the main object we will be interacting with in pandas. It\n",
    "represents a table of data.  The other file formats all start with\n",
    "`pd.read_{format}`.  Note that we can provide the URL to the dataset, rather\n",
    "than download it beforehand.\n",
    "\n",
    "We can write out the dataset using `dataframe.to_{format}(<filename)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.to_csv('titanic_copy.csv', index=False)  # we set index to False to prevent pandas from storing the row names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you can not connect to the internet, you can run the command below to load\n",
    "this locally stored titanic dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic = pd.read_csv('09_pandas/titanic.csv')\n",
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the titanic dataset was also available to us as one of the standard\n",
    "datasets included with seaborn. We could load it from there using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.load_dataset('titanic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Dataframes` can also be created from other python objects, using\n",
    "`pd.DataFrame.from_{other type}`. The most useful of these is `from_dict`,\n",
    "which converts a mapping of the columns to a pandas `DataFrame` (i.e., table)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame.from_dict({\n",
    "    'random numbers': np.random.rand(5),\n",
    "    'sequence (int)': np.arange(5),\n",
    "    'sequence (float)': np.linspace(0, 5, 5),\n",
    "    'letters': list('abcde'),\n",
    "    'constant_value': 'same_value'\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many applications (e.g., ICA, machine learning input) you might want to\n",
    "extract your data as a numpy array. The underlying numpy array can be accessed\n",
    "using the `values` attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the type of the returned array is the most common type (in this case\n",
    "object). If you just want the numeric parts of the table you can use\n",
    "`select_dtypes`, which selects specific columns based on their dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic.select_dtypes(include=np.number).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the numpy array has no information on the column names or row indices.\n",
    "Alternatively, when you want to include the categorical variables in your later\n",
    "analysis (e.g., for machine learning), you can extract dummy variables using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.get_dummies(titanic)"
   ]
  },
  {
   "cell_type": "markdown",