pandas.ipynb 36.8 KB
Newer Older
1
2
3
4
5
6
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
7
8
    "# Pandas\n",
    "\n",
9
10
    "Follow along online at: https://git.fmrib.ox.ac.uk/fsl/pytreat-practicals-2020/-/blob/master/talks/pandas/pandas.ipynb\n",
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
11
12
    "Pandas is a data analysis library focused on the cleaning and exploration of\n",
    "tabular data.\n",
13
14
15
16
    "\n",
    "Some useful links are:\n",
    "- [main website](https://pandas.pydata.org)\n",
    "- [documentation](http://pandas.pydata.org/pandas-docs/stable/)<sup>1</sup>\n",
Paul McCarthy's avatar
Paul McCarthy committed
17
18
    "- [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/)<sup>1</sup> by\n",
    "  Jake van der Plas\n",
19
    "- [List of Pandas tutorials](https://pandas.pydata.org/pandas-docs/stable/getting_started/tutorials.html)\n",
20
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
21
22
    "<sup>1</sup> This tutorial borrows heavily from the pandas documentation and\n",
    "the Python Data Science Handbook"
23
24
25
26
27
28
29
30
31
32
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline\n",
    "import pandas as pd  # pd is the usual abbreviation for pandas\n",
Paul McCarthy's avatar
Paul McCarthy committed
33
    "import matplotlib.pyplot as plt # matplotlib for plotting\n",
34
35
36
37
38
39
40
41
42
43
44
    "import seaborn as sns  # seaborn is the main plotting library for Pandas\n",
    "import statsmodels.api as sm  # statsmodels fits linear models to pandas data\n",
    "import statsmodels.formula.api as smf\n",
    "from IPython.display import Image\n",
    "sns.set()  # use the prettier seaborn plotting settings rather than the default matplotlib one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
45
46
47
48
49
50
51
52
53
    "> We will mostly be using `seaborn` instead of `matplotlib` for\n",
    "> visualisation. But `seaborn` is actually an extension to `matplotlib`, so we\n",
    "> are still using the latter under the hood.\n",
    "\n",
    "## Loading in data\n",
    "\n",
    "Pandas supports a wide range of I/O tools to load from text files, binary files,\n",
    "and SQL databases. You can find a table with all formats\n",
    "[here](http://pandas.pydata.org/pandas-docs/stable/io.html)."
54
55
56
57
58
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
59
   "metadata": {},
60
61
62
63
64
65
66
67
68
69
   "outputs": [],
   "source": [
    "titanic = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv')\n",
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
70
71
72
73
74
75
    "This loads the data into a\n",
    "[`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)\n",
    "object, which is the main object we will be interacting with in pandas. It\n",
    "represents a table of data.  The other file formats all start with\n",
    "`pd.read_{format}`.  Note that we can provide the URL to the dataset, rather\n",
    "than download it beforehand.\n",
76
    "\n",
77
    "We can write out the dataset using `dataframe.to_{format}(<filename>)`:"
78
79
80
81
82
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
83
   "metadata": {},
84
85
86
87
88
89
90
91
92
   "outputs": [],
   "source": [
    "titanic.to_csv('titanic_copy.csv', index=False)  # we set index to False to prevent pandas from storing the row names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
93
94
    "If you can not connect to the internet, you can run the command below to load\n",
    "this locally stored titanic dataset"
95
96
97
98
99
100
101
102
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
103
    "titanic = pd.read_csv('titanic.csv')\n",
104
105
106
107
108
109
110
    "titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
111
112
    "Note that the titanic dataset was also available to us as one of the standard\n",
    "datasets included with seaborn. We could load it from there using"
113
114
115
116
117
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
118
   "metadata": {},
119
120
121
122
123
124
125
126
127
   "outputs": [],
   "source": [
    "sns.load_dataset('titanic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
128
129
130
    "`Dataframes` can also be created from other python objects, using\n",
    "`pd.DataFrame.from_{other type}`. The most useful of these is `from_dict`,\n",
    "which converts a mapping of the columns to a pandas `DataFrame` (i.e., table)."
131
132
133
134
135
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
136
   "metadata": {},
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
   "outputs": [],
   "source": [
    "pd.DataFrame.from_dict({\n",
    "    'random numbers': np.random.rand(5),\n",
    "    'sequence (int)': np.arange(5),\n",
    "    'sequence (float)': np.linspace(0, 5, 5),\n",
    "    'letters': list('abcde'),\n",
    "    'constant_value': 'same_value'\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
152
153
    "For many applications (e.g., ICA, machine learning input) you might want to\n",
    "extract your data as a numpy array. The underlying numpy array can be accessed\n",
154
    "using the `to_numpy` method"
155
156
157
158
159
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
160
   "metadata": {},
161
162
   "outputs": [],
   "source": [
163
    "titanic.to_numpy()"
164
165
166
167
168
169
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
170
171
172
    "Note that the type of the returned array is the most common type (in this case\n",
    "object). If you just want the numeric parts of the table you can use\n",
    "`select_dtypes`, which selects specific columns based on their dtype:"
173
174
175
176
177
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
178
   "metadata": {},
179
180
   "outputs": [],
   "source": [
181
    "titanic.select_dtypes(include=np.number).to_numpy()"
182
183
184
185
186
187
188
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the numpy array has no information on the column names or row indices.\n",
Paul McCarthy's avatar
Paul McCarthy committed
189
190
    "Alternatively, when you want to include the categorical variables in your later\n",
    "analysis (e.g., for machine learning), you can extract dummy variables using:"
191
192
193
194
195
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
196
   "metadata": {},
197
198
199
200
201
202
203
204
205
   "outputs": [],
   "source": [
    "pd.get_dummies(titanic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
206
207
208
209
210
211
    "## Accessing parts of the data\n",
    "\n",
    "[Documentation on indexing](http://pandas.pydata.org/pandas-docs/stable/indexing.html)\n",
    "\n",
    "### Selecting columns by name\n",
    "\n",
212
213
214
215
216
217
    "Single columns can be selected using the normal python indexing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
218
   "metadata": {},
219
220
221
222
223
224
225
226
227
   "outputs": [],
   "source": [
    "titanic['embark_town']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
228
229
    "If the column names are simple strings (not required) we can also access it\n",
    "directly as an attribute"
230
231
232
233
234
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
235
   "metadata": {},
236
237
238
239
240
241
242
243
244
   "outputs": [],
   "source": [
    "titanic.embark_town"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
245
246
247
248
249
    "Note that this returns a pandas\n",
    "[`Series`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html)\n",
    "rather than a `DataFrame` object. A `Series` is simply a 1-dimensional array\n",
    "representing a single column.  Multiple columns can be returned by providing a\n",
    "list of columns names. This will return a `DataFrame`:"
250
251
252
253
254
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
255
   "metadata": {},
256
257
258
259
260
261
262
263
264
   "outputs": [],
   "source": [
    "titanic[['class', 'alive']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
265
266
267
    "Note that you have to provide a list here (square brackets). If you provide a\n",
    "tuple (round brackets) pandas will think you are trying to access a single\n",
    "column that has that tuple as a name:"
268
269
270
271
272
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
273
   "metadata": {},
274
275
276
277
278
279
280
281
282
   "outputs": [],
   "source": [
    "titanic[('class', 'alive')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
283
284
285
286
287
288
289
290
291
    "In this case there is no column called `('class', 'alive')` leading to an\n",
    "error.  Later on we will see some uses to having columns named like this.\n",
    "\n",
    "### Indexing rows by name or integer\n",
    "\n",
    "Individual rows can be accessed based on their name (i.e., the index) or integer\n",
    "(i.e., which row it is in). In our current table this will give the same\n",
    "results. To ensure that these are different, let's sort our titanic dataset\n",
    "based on the passenger fare:"
292
293
294
295
296
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
297
   "metadata": {},
298
299
300
301
302
303
304
305
306
307
   "outputs": [],
   "source": [
    "titanic_sorted = titanic.sort_values('fare')\n",
    "titanic_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
308
309
    "Note that the re-sorting did not change the values in the index (i.e., left-most\n",
    "column).\n",
310
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
311
    "We can select the first row of this newly sorted table using `iloc`"
312
313
314
315
316
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
317
   "metadata": {},
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
   "outputs": [],
   "source": [
    "titanic_sorted.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can select the row with the index 0 using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
333
   "metadata": {},
334
335
336
337
338
339
340
341
342
   "outputs": [],
   "source": [
    "titanic_sorted.loc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
343
344
    "Note that this gives the same passenger as the first row of the initial table\n",
    "before sorting"
345
346
347
348
349
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
350
   "metadata": {},
351
352
353
354
355
356
357
358
359
   "outputs": [],
   "source": [
    "titanic.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
360
361
    "Another common way to access the first or last N rows of a table is using the\n",
    "head/tail methods"
362
363
364
365
366
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
367
   "metadata": {},
368
369
370
371
372
373
374
375
   "outputs": [],
   "source": [
    "titanic_sorted.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
376
   "metadata": {},
377
378
379
380
381
382
383
384
385
   "outputs": [],
   "source": [
    "titanic_sorted.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
386
387
    "Note that nearly all methods in pandas return a new `Dataframe`, which means\n",
    "that we can easily call another method on them"
388
389
390
391
392
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
393
   "metadata": {},
394
395
396
397
398
399
400
401
   "outputs": [],
   "source": [
    "titanic_sorted.tail(10).head(5)  # select the first 5 of the last 10 passengers in the database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
402
   "metadata": {},
403
404
405
406
407
408
409
410
411
   "outputs": [],
   "source": [
    "titanic_sorted.iloc[-10:-5]  # alternative way to get the same passengers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
412
413
414
    "**Exercise**: use sorting and tail/head or indexing to find the 10 youngest\n",
    "passengers on the titanic. Try to do this on a single line by chaining calls\n",
    "to the titanic `DataFrame` object"
415
416
417
418
419
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
420
   "metadata": {},
421
422
423
424
425
426
427
428
429
   "outputs": [],
   "source": [
    "titanic.sort_values..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
430
431
    "### Indexing rows by value\n",
    "\n",
432
433
434
435
436
437
    "One final way to select specific columns is by their value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
438
   "metadata": {},
439
440
441
442
443
444
445
446
   "outputs": [],
   "source": [
    "titanic[titanic.sex == 'female']  # selects all females"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
447
   "metadata": {},
448
449
450
451
452
453
454
455
456
457
   "outputs": [],
   "source": [
    "# select all passengers older than 60 who departed from Southampton\n",
    "titanic[(titanic.age > 60) & (titanic['embark_town'] == 'Southampton')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
458
459
460
461
462
    "Note that this required typing `titanic` quite often. A quicker way to get the\n",
    "same result is using the `query` method, which is described in detail\n",
    "[here](http://pandas.pydata.org/pandas-docs/stable/indexing.html#the-query-method)\n",
    "(note that using the `query` method is also faster and uses a lot less\n",
    "memory).\n",
463
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
464
465
    "> You may have trouble using the `query` method with columns which have\n",
    "a name that cannot be used as a Python identifier."
466
467
468
469
470
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
471
   "metadata": {},
472
473
474
475
476
   "outputs": [],
   "source": [
    "titanic.query('(age > 60) & (embark_town == \"Southampton\")')"
   ]
  },
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When selecting a categorical multiple options from a categorical values you \n",
    "might want to use `isin`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic[titanic['class'].isin(['First','Second'])]"
   ]
  },
494
495
496
497
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
498
499
    "Particularly useful when selecting data like this is the `isna` method which\n",
    "finds all missing data"
500
501
502
503
504
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
505
   "metadata": {},
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
   "outputs": [],
   "source": [
    "titanic[~titanic.age.isna()]  # select first few passengers whose age is not N/A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This removing of missing numbers is so common that it has is own method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
521
   "metadata": {},
522
523
524
525
526
527
528
529
   "outputs": [],
   "source": [
    "titanic.dropna()  # drops all passengers that have some datapoint missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
530
   "metadata": {},
531
532
533
534
535
536
537
538
539
   "outputs": [],
   "source": [
    "titanic.dropna(subset=['age', 'fare'])  # Only drop passengers with missing ages or fares"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
540
541
542
    "**Exercise**: use sorting, indexing by value, `dropna` and `tail`/`head` or\n",
    "indexing to find the 10 oldest female passengers on the titanic. Try to do\n",
    "this on a single line by chaining calls to the titanic `DataFrame` object"
543
544
545
546
547
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
548
   "metadata": {},
549
550
551
552
553
554
555
556
557
   "outputs": [],
   "source": [
    "titanic..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
558
    "## Plotting the data\n",
559
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
560
    "Before we start analyzing the data, let's play around with visualizing it.\n",
561
562
563
564
565
566
    "Pandas does have some basic built-in plotting options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
567
   "metadata": {},
568
569
570
571
572
573
574
575
   "outputs": [],
   "source": [
    "titanic.fare.hist(bins=20, log=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
576
   "metadata": {},
577
578
579
580
581
582
583
584
585
   "outputs": [],
   "source": [
    "titanic.age.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
586
587
588
589
590
    "To plot all variables simply call `plot` or `hist` on the full dataframe\n",
    "rather than a single Series (i.e., column). You might want to set `subplots=True`\n",
    "to plot each variable in a different subplot.\n",
    "\n",
    "Individual Series are essentially 1D arrays, so we can use them as such in\n",
Paul McCarthy's avatar
Paul McCarthy committed
591
    "`matplotlib`"
592
593
594
595
596
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
597
   "metadata": {},
598
599
600
601
602
603
604
605
606
   "outputs": [],
   "source": [
    "plt.scatter(titanic.age, titanic.fare)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
607
608
609
610
611
612
613
614
615
    "However, for most purposes much nicer plots can be obtained using\n",
    "[Seaborn](https://seaborn.pydata.org). Seaborn has support to produce plots\n",
    "showing the\n",
    "[univariate](https://seaborn.pydata.org/tutorial/distributions.html#plotting-univariate-distributions)\n",
    "or\n",
    "[bivariate](https://seaborn.pydata.org/tutorial/distributions.html#plotting-bivariate-distributions)\n",
    "distribution of data in a single or a grid of plots.  Most of the seaborn\n",
    "plotting functions expect to get a pandas `DataFrame` (although they will work\n",
    "with Numpy arrays as well). So we can plot age vs. fare like:"
616
617
618
619
620
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
621
   "metadata": {},
622
623
624
625
626
627
628
629
630
   "outputs": [],
   "source": [
    "sns.jointplot('age', 'fare', data=titanic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
631
632
633
    "**Exercise**: check the documentation from `sns.jointplot` (hover the mouse\n",
    "over the text `jointplot` and press shift-tab) to find out how to turn the\n",
    "scatter plot into a density (kde) map"
634
635
636
637
638
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
639
   "metadata": {},
640
641
642
643
644
645
646
647
648
   "outputs": [],
   "source": [
    "sns.jointplot('age', 'fare', data=titanic, ...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
649
650
    "Here is just a brief example of how we can use multiple columns to illustrate\n",
    "the data in more detail"
651
652
653
654
655
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
656
   "metadata": {},
657
658
659
660
661
662
663
664
665
666
   "outputs": [],
   "source": [
    "sns.relplot(x='age', y='fare', col='class', hue='sex', data=titanic,\n",
    "           col_order=('First', 'Second', 'Third'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
667
668
669
670
    "**Exercise**: Split the plot above into two rows with the first row including\n",
    "the passengers who survived and the second row those who did not (you might\n",
    "have to check the documentation again by using shift-tab while overing the\n",
    "mouse over `relplot`)"
671
672
673
674
675
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
676
   "metadata": {},
677
678
679
680
681
682
683
684
685
686
   "outputs": [],
   "source": [
    "sns.relplot(x='age', y='fare', col='class', hue='sex', data=titanic,\n",
    "           col_order=('First', 'Second', 'Third')...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
687
688
689
690
    "One of the nice thing of Seaborn is how easy it is to update how these plots\n",
    "look. You can read more about that\n",
    "[here](https://seaborn.pydata.org/tutorial/aesthetics.html). For example, to\n",
    "increase the font size to get a plot more approriate for a talk, you can use:"
691
692
693
694
695
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
696
   "metadata": {},
697
698
699
   "outputs": [],
   "source": [
    "sns.set_context('talk')\n",
Paul McCarthy's avatar
Paul McCarthy committed
700
    "sns.violinplot(x='class', y='age', hue='sex', data=titanic, split=True,\n",
701
702
703
704
705
706
707
    "               order=('First', 'Second', 'Third'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
708
709
710
711
712
    "## Summarizing the data (mean, std, etc.)\n",
    "\n",
    "There are a large number of built-in methods to summarize the observations in\n",
    "a Pandas `DataFrame`. Most of these will return a `Series` with the columns\n",
    "names as index:"
713
714
715
716
717
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
718
   "metadata": {},
719
720
721
722
723
724
725
726
   "outputs": [],
   "source": [
    "titanic.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
727
   "metadata": {},
728
729
730
731
732
733
734
735
736
   "outputs": [],
   "source": [
    "titanic.quantile(0.75)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
737
738
    "One very useful one is `describe`, which gives an overview of many common\n",
    "summary measures"
739
740
741
742
743
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
744
   "metadata": {},
745
746
747
748
749
   "outputs": [],
   "source": [
    "titanic.describe()"
   ]
  },
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a more detailed exploration of the data, you might want to check \n",
    "[pandas_profiliing](https://pandas-profiling.github.io/pandas-profiling/docs/)\n",
    "(not installed in fslpython, so the following will not run in fslpython):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pandas_profiling import ProfileReport\n",
    "profile = ProfileReport(titanic, title='Titanic Report', html={'style':{'full_width':True}})\n",
    "profile.to_widgets()"
   ]
  },
770
771
772
773
774
775
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that non-numeric columns are ignored when summarizing data in this way.\n",
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
776
777
    "We can also define our own functions to apply to the columns (in this case we\n",
    "have to explicitly set the data types)."
778
779
780
781
782
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
783
   "metadata": {},
784
785
786
787
788
   "outputs": [],
   "source": [
    "def mad(series):\n",
    "    \"\"\"\n",
    "    Computes the median absolute deviatation (MAD)\n",
Paul McCarthy's avatar
Paul McCarthy committed
789
    "\n",
790
791
792
793
794
795
796
797
798
799
800
801
    "    This is a outlier-resistant measure of the standard deviation\n",
    "    \"\"\"\n",
    "    no_nan = series.dropna()\n",
    "    return np.median(abs(no_nan - np.nanmedian(no_nan)))\n",
    "\n",
    "titanic.select_dtypes(np.number).apply(mad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
802
803
    "We can also provide multiple functions to the `apply` method (note that\n",
    "functions can be provided as strings)"
804
805
806
807
808
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
809
   "metadata": {},
810
811
812
813
814
815
816
817
818
   "outputs": [],
   "source": [
    "titanic.select_dtypes(np.number).apply(['mean', np.median, np.std, mad])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
819
820
821
822
823
824
825
826
    "### Grouping by\n",
    "\n",
    "One of the more powerful features of is `groupby`, which splits the dataset on\n",
    "a categorical variable. The book contains a clear tutorial on that feature\n",
    "[here](https://jakevdp.github.io/PythonDataScienceHandbook/03.08-aggregation-and-grouping.html). You\n",
    "can check the pandas documentation\n",
    "[here](http://pandas.pydata.org/pandas-docs/stable/groupby.html) for a more\n",
    "formal introduction. One simple use is just to put it into a loop"
827
828
829
830
831
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
832
   "metadata": {},
833
834
835
836
837
838
839
840
841
842
   "outputs": [],
   "source": [
    "for cls, part_table in titanic.groupby('class'):\n",
    "    print(f'Mean fare in {cls.lower()} class: {part_table.fare.mean()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
843
844
845
    "However, it is more often combined with one of the aggregation functions\n",
    "discussed above as illustrated in this figure from the [Python data science\n",
    "handbook](https://jakevdp.github.io/PythonDataScienceHandbook/06.00-figure-code.html#Split-Apply-Combine)\n",
846
    "\n",
847
    "![group by image](group_by.png)"
848
849
850
851
852
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
853
   "metadata": {},
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
   "outputs": [],
   "source": [
    "titanic.groupby('class').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also group by multiple variables at once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
869
   "metadata": {},
870
871
872
873
874
875
876
877
878
   "outputs": [],
   "source": [
    "titanic.groupby(['class', 'survived']).mean()  # as always in pandas supply multiple column names as lists, not tuples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
879
880
    "When grouping it can help to use the `cut` method to split a continuous variable\n",
    "into a categorical one"
881
882
883
884
885
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
886
   "metadata": {},
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
   "outputs": [],
   "source": [
    "titanic.groupby(['class', pd.cut(titanic.age, bins=(0, 18, 50, np.inf))]).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `aggregate` method to apply a different function to each series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
902
   "metadata": {},
903
904
905
906
907
908
909
910
911
   "outputs": [],
   "source": [
    "titanic.groupby(['class', 'survived']).aggregate((np.median, mad))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
912
913
914
915
916
917
    "Note that both the index (on the left) and the column names (on the top) now\n",
    "have multiple levels. Such a multi-level index is referred to as `MultiIndex`.\n",
    "This does complicate selecting specific columns/rows. You can read more of using\n",
    "`MultiIndex` [here](http://pandas.pydata.org/pandas-docs/stable/advanced.html).\n",
    "The short version is that columns can be selected using direct indexing (as\n",
    "discussed above)"
918
919
920
921
922
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
923
   "metadata": {},
924
925
926
927
928
929
930
931
   "outputs": [],
   "source": [
    "df_full = titanic.groupby(['class', 'survived']).aggregate((np.median, mad))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
932
   "metadata": {},
933
934
935
936
937
938
939
940
   "outputs": [],
   "source": [
    "df_full[('age', 'median')]  # selects median age column; note that the round brackets are optional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
941
   "metadata": {},
942
943
944
945
946
947
948
949
950
   "outputs": [],
   "source": [
    "df_full['age']  # selects both age columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
951
952
    "Remember that indexing based on the index was done through `loc`. The rest is\n",
    "the same as for the columns above"
953
954
955
956
957
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
958
   "metadata": {},
959
960
961
962
963
964
965
966
   "outputs": [],
   "source": [
    "df_full.loc[('First', 0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
967
   "metadata": {},
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
   "outputs": [],
   "source": [
    "df_full.loc['First']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More advanced use of the `MultiIndex` is possible through `xs`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
983
   "metadata": {},
984
985
986
987
988
989
990
991
992
   "outputs": [],
   "source": [
    "df_full.xs(0, level='survived') # selects all the zero's from the survived index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
Paul McCarthy's avatar
Paul McCarthy committed
993
   "outputs": [],
994
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
995
    "df_full.xs('mad', axis=1, level=1) # selects mad from the second level in the columns (i.e., axis=1)"
996
997
998
999
1000
1001
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1002
1003
1004
1005
    "## Reshaping tables\n",
    "\n",
    "If we were interested in how the survival rate depends on the class and sex of\n",
    "the passengers we could simply use a groupby:"
1006
1007
1008
1009
1010
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1011
   "metadata": {},
1012
1013
1014
1015
1016
1017
1018
1019
1020
   "outputs": [],
   "source": [
    "titanic.groupby(['class', 'sex']).survived.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1021
1022
1023
1024
1025
    "However, this single-column table is difficult to read. The reason for this is\n",
    "that the indexing is multi-leveled (called `MultiIndex` in pandas), while there\n",
    "is only a single column. We would like to move one of the levels in the index to\n",
    "the columns. This can be done using `stack`/`unstack`:\n",
    "\n",
1026
1027
1028
1029
1030
1031
1032
    "- `unstack`: Moves one levels in the index to the columns\n",
    "- `stack`: Moves one of levels in the columns to the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1033
   "metadata": {},
1034
1035
1036
1037
1038
1039
1040
1041
1042
   "outputs": [],
   "source": [
    "titanic.groupby(['class', 'sex']).survived.mean().unstack('sex')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1043
1044
1045
1046
1047
1048
1049
1050
1051
    "The former table, where the different groups are defined in different rows, is\n",
    "often referred to as long-form. After unstacking the table is often referred to\n",
    "as wide-form as the different group (sex in this case) is now represented as\n",
    "different columns. In pandas some operations are easier on long-form tables\n",
    "(e.g., `groupby`) while others require wide_form tables (e.g., making scatter\n",
    "plots of two variables). You can go back and forth using `unstack` or `stack` as\n",
    "illustrated above, but as this is a crucial part of pandas there are many\n",
    "alternatives, such as `pivot_table`, `melt`, and `wide_to_long`, which we will\n",
    "discuss below.\n",
1052
1053
1054
1055
1056
1057
1058
    "\n",
    "We can prettify the table further using seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1059
   "metadata": {},
1060
1061
   "outputs": [],
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1062
    "ax = sns.heatmap(titanic.groupby(['class', 'sex']).survived.mean().unstack('sex'),\n",
1063
1064
1065
1066
1067
1068
1069
1070
    "                 annot=True)\n",
    "ax.set_title('survival rate')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1071
1072
1073
1074
1075
1076
    "Note that there are also many ways to produce prettier tables in pandas (e.g.,\n",
    "color all the negative values). This is documented\n",
    "[here](http://pandas.pydata.org/pandas-docs/stable/style.html).\n",
    "\n",
    "Because this stacking/unstacking is fairly common after a groupby operation,\n",
    "there is a shortcut for it: `pivot_table`"
1077
1078
1079
1080
1081
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1082
   "metadata": {},
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
   "outputs": [],
   "source": [
    "titanic.pivot_table('survived', 'class', 'sex')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual in pandas, where we can also provide multiple column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1098
   "metadata": {},
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
   "outputs": [],
   "source": [
    "sns.heatmap(titanic.pivot_table('survived', ['class', 'embark_town'], ['sex', pd.cut(titanic.age, (0, 18, np.inf))]), annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also change the function to be used to aggregate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1114
   "metadata": {},
1115
1116
   "outputs": [],
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1117
    "sns.heatmap(titanic.pivot_table('survived', ['class', 'embark_town'], ['sex', pd.cut(titanic.age, (0, 18, np.inf))],\n",
1118
1119
1120
1121
1122
1123
1124
    "                                aggfunc='count'), annot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1125
1126
    "As in `groupby` the aggregation function can be a string of a common aggregation\n",
    "function, or any function that should be applied.\n",
1127
1128
1129
1130
1131
1132
1133
    "\n",
    "We can even apply different aggregate functions to different columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1134
   "metadata": {},
1135
1136
   "outputs": [],
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1137
    "titanic.pivot_table(index='class', columns='sex',\n",
1138
1139
1140
1141
1142
1143
1144
    "                    aggfunc={'survived': 'count', 'fare': np.mean}) # compute number of survivors and mean fare\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1145
1146
1147
1148
    "The opposite of `pivot_table` is `melt`. This can be used to change a wide-form\n",
    "table into a long-form table. This is not particularly useful on the titanic\n",
    "dataset, so let's create a new table where this might be useful. Let's say we\n",
    "have a dataset listing the FA and MD values in various WM tracts:"
1149
1150
1151
1152
1153
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1154
   "metadata": {},
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
   "outputs": [],
   "source": [
    "tracts = ('Corpus callosum', 'Internal capsule', 'SLF', 'Arcuate fasciculus')\n",
    "df_wide = pd.DataFrame.from_dict(dict({'subject': list('ABCDEFGHIJ')}, **{\n",
    "    f'FA({tract})': np.random.rand(10) for tract in tracts }, **{\n",
    "    f'MD({tract})': np.random.rand(10) * 1e-3 for tract in tracts\n",
    "}))\n",
    "df_wide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1169
1170
1171
1172
1173
    "This wide-form table (i.e., all the information is in different columns) makes\n",
    "it hard to select just all the FA values or only the values associated with the\n",
    "SLF. For this it would be easier to list all the values in a single column.\n",
    "Most of the tools discussed above (e.g., `group_by` or `seaborn` plotting) work\n",
    "better with long-form data, which we can obtain from `melt`:"
1174
1175
1176
1177
1178
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1179
   "metadata": {},
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
   "outputs": [],
   "source": [
    "df_long = df_wide.melt('subject', var_name='measurement', value_name='dti_value')\n",
    "df_long.head(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1190
1191
1192
1193
1194
1195
    "We can see that `melt` took all the columns (we could also have specified a\n",
    "specific sub-set) and returned each measurement as a seperate row. We probably\n",
    "want to seperate the measurement column into the measurement type (FA or MD) and\n",
    "the tract name. Many string manipulation function are available in the\n",
    "`DataFrame` object under `DataFrame.str`\n",
    "([tutorial](http://pandas.pydata.org/pandas-docs/stable/text.html))"
1196
1197
1198
1199
1200
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1201
   "metadata": {},
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
   "outputs": [],
   "source": [
    "df_long['variable'] = df_long.measurement.str.slice(0, 2)  # first two letters correspond to FA or MD\n",
    "df_long['tract'] = df_long.measurement.str.slice(3, -1)  # fourth till the second-to-last letter correspond to the tract\n",
    "df_long.head(12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1213
    "Finally we probably do want the FA and MD variables as different columns.\n",
1214
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
1215
1216
    "**Exercise**: Use `pivot_table` or `stack`/`unstack` to create a column for MD\n",
    "and FA."
1217
1218
1219
1220
1221
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1222
   "metadata": {},
1223
1224
1225
1226
1227
1228
1229
1230
1231
   "outputs": [],
   "source": [
    "df_unstacked = df_long."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1232
1233
    "We can now use the tools discussed above to visualize the table (`seaborn`) or\n",
    "to group the table based on tract (`groupby` or `pivot_table`)."
1234
1235
1236
1237
1238
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1239
   "metadata": {},
1240
1241
1242
1243
1244
1245
1246
1247
1248
   "outputs": [],
   "source": [
    "# feel free to analyze this random data in more detail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1249
1250
1251
1252
    "In general pandas is better at handling long-form than wide-form data, although\n",
    "for better visualization of the data an intermediate format is often best. One\n",
    "exception is calculating a covariance (`DataFrame.cov`) or correlation\n",
    "(`DataFrame.corr`) matrices which computes the correlation between each column:"
1253
1254
1255
1256
1257
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1258
   "metadata": {},
1259
1260
1261
1262
1263
1264
1265
1266
1267
   "outputs": [],
   "source": [
    "sns.heatmap(df_wide.corr(), cmap=sns.diverging_palette(240, 10, s=99, n=300), )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
    "## Linear fitting (`statsmodels`)\n",
    "\n",
    "Linear fitting between the different columns is available through the\n",
    "[`statsmodels`](https://www.statsmodels.org/stable/index.html) library. A nice\n",
    "way to play around with a wide variety of possible models is to use R-style\n",
    "functions. The usage of the functions in `statsmodels` is described\n",
    "[here](https://www.statsmodels.org/dev/example_formulas.html). You can find a\n",
    "more detailed description of the R-style functions\n",
    "[here](https://patsy.readthedocs.io/en/latest/formulas.html#the-formula-\n",
    "language).\n",
1278
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
1279
1280
1281
1282
    "In short these functions describe the linear model as a string. For example,\n",
    "`\"y ~ x + a + x * a\"` fits the variable `y` as a function of `x`, `a`, and the\n",
    "interaction between `x` and `a`. The intercept is included by default (you can\n",
    "add `\"+ 0\"` to remove it)."
1283
1284
1285
1286
1287
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1288
   "metadata": {},
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
   "outputs": [],
   "source": [
    "result = smf.logit('survived ~ age + sex + age * sex', data=titanic).fit()\n",
    "print(result.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Paul McCarthy's avatar
Paul McCarthy committed
1299
1300
    "Note that `statsmodels` understands categorical variables and automatically\n",
    "replaces them with dummy variables.\n",
1301
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
1302
1303
1304
    "Above we used logistic regression, which is appropriate for the binary\n",
    "survival rate. A wide variety of linear models are available. Let's try a GLM,\n",
    "but assume that the fare is drawn from a Gamma distribution:"
1305
1306
1307
1308
1309
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
Paul McCarthy's avatar
Paul McCarthy committed
1310
   "metadata": {},
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
   "outputs": [],
   "source": [
    "age_dmean = titanic.age - titanic.age.mean()\n",
    "result = smf.glm('fare ~ age_dmean + embark_town', data=titanic).fit()\n",
    "print(result.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cherbourg passengers clearly paid a lot more...\n",
    "\n",
    "\n",
Paul McCarthy's avatar
Paul McCarthy committed
1325
1326
1327
1328
1329
1330
1331
    "Note that we did not actually add the `age_dmean` to the\n",
    "`DataFrame`. `statsmodels` (or more precisely the underlying\n",
    "[patsy](https://patsy.readthedocs.io/en/latest/) library) automatically\n",
    "extracted this from our environment. This can lead to confusing behaviour...\n",
    "\n",
    "# More reading\n",
    "\n",
1332
    "Other useful features\n",
Paul McCarthy's avatar
Paul McCarthy committed
1333
    "\n",
1334
1335
    "- [Concatenating and merging tables](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/08_combine_dataframes.html)\n",
    "- [Lots of time series support](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/09_timeseries.html)\n",
Paul McCarthy's avatar
Paul McCarthy committed
1336
1337
1338
    "- [Rolling Window\n",
    "  functions](http://pandas.pydata.org/pandas-docs/stable/computation.html#window-\n",
    "  functions) for after you have meaningfully sorted your data\n",
1339
1340
1341
1342
    "- and much, much more"
   ]
  }
 ],
Paul McCarthy's avatar
Paul McCarthy committed
1343
 "metadata": {},
1344
 "nbformat": 4,
1345
 "nbformat_minor": 4
1346
}