09_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is a data analysis library focussed on the cleaning and exploration of tabular data.\n",
    "\n",
    "Some useful links are:\n",
    "- [main website](https://pandas.pydata.org)\n",
    "- [documentation](http://pandas.pydata.org/pandas-docs/stable/)<sup>1</sup>\n",
    "- [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)<sup>1</sup> by Jake van der Plas\n",
    "\n",
    "<sup>1</sup> This tutorial borrows heavily from the pandas documentation and the Python Data Science Handbook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import pandas as pd  # pd is the usual abbreviation for pandas\n",
    "import seaborn as sns  # seaborn is the main plotting library for Pandas\n",
    "import statsmodels.api as sm  # statsmodels fits linear models to pandas data\n",
    "import statsmodels.formula.api as smf\n",
    "from IPython.display import Image\n",
    "sns.set()  # use the prettier seaborn plotting settings rather than the default matplotlib one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading in data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas supports a wide range of I/O tools to load from text files, binary files, and SQL databases. You can find a table with all formats [here](http://pandas.pydata.org/pandas-docs/stable/io.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titanic = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv')\n",
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This loads the data into a [DataFrame](https://pandas.pydata.org/pandas-docs/version/0.21/generated/pandas.DataFrame.html) object, which is the main object we will be interacting with in pandas. It represents a table of data.\n",
    "\n",
    "The other file formats all start with `pd.read_{format}`.  Note that we can provide the URL to the dataset, rather than download it beforehand.\n",
    "\n",
    "We can write out the dataset using `dataframe.to_{format}(<filename)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titanic.to_csv('titanic_copy.csv', index=False)  # we set index to False to prevent pandas from storing the row names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you can not connect to the internet, you can run the command below to load this locally stored titanic dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic = pd.read_csv('09_pandas/titanic.csv')\n",
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the titanic dataset was also available to us as one of the standard datasets included with seaborn. We could load it from there using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sns.load_dataset('titanic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataframes can also be created from other python objects, using pd.DataFrame.from_{other type}. The most useful of these is from_dict, which converts a mapping of the columns to a pandas DataFrame (i.e., table).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame.from_dict({\n",
    "    'random numbers': np.random.rand(5),\n",
    "    'sequence (int)': np.arange(5),\n",
    "    'sequence (float)': np.linspace(0, 5, 5),\n",
    "    'letters': list('abcde'),\n",
    "    'constant_value': 'same_value'\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many applications (e.g., ICA, machine learning input) you might want to extract your data as a numpy array. The underlying numpy array can be accessed using the `values` attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titanic.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the type of the returned array is the most common type (in this case object). If you just want the numeric parts of the table you can use `select_dtype`, which selects specific columns based on their dtype:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titanic.select_dtypes(include=np.number).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the numpy array has no information on the column names or row indices.\n",
    "\n",
    "Alternatively, when you want to include the categorical variables in your later analysis (e.g., for machine learning), you can extract dummy variables using: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.get_dummies(titanic)"
   ]