{ "cells": [ { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0, "toc": true }, "source": [ "

Table of Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0 }, "source": [ "# Reading & Summarizing CSV Data\n", "\n", "Credit: the notebooks in this folder are lightly modified versions of work by Jennifer Walker presented\n", "at the EOAS python workshop in October, 2018: https://github.com/jenfly/eoas-python\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "import pandas" ] }, { "cell_type": "markdown", "metadata": { "lines_to_next_cell": 0 }, "source": [ "## Learning objectives\n", "\n", "* Learn how to read, process and write data in csv/xlsx/tabular format using pandas\n", "\n", "* In parts 2 and parts 3:\n", "\n", " * Learn how to download tabular data from websites with a \"restful api\":\n", " - https://stackoverflow.com/questions/671118/what-exactly-is-restful-programming\n", "\n", " * Learn how to clean data by filtering missing values, renaming columns, and writing out\n", " processed files for further work" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Folder setup\n", "\n", "As our project grows more complicated, it's good to have a central\n", "module that keeps track of important files and sets your scripts\n", "up so that they can import functions and classes from you modules.\n", "If you were planning to distribute your project using conda, then\n", "you would need to write an installation script, which is a fair\n", "amount of work. At this stage, it's easier and more flexible to\n", "store that information in a file that travels along with your notebook.\n", "We set the \"context\" for this notebook by importing:\n", "[context_pandas1.py](https://github.com/phaustin/eosc213_students/blob/master/notebooks/pandas/context_pandas1.py)\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "lines_to_next_cell": 0 }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "in context_pandas1.py, setting root_dir to \n", "/Users/phil/repos/eosc213_students/notebooks/pandas\n", "setting data_dir to \n", "/Users/phil/repos/eosc213_students/notebooks/pandas/data\n", "\n" ] } ], "source": [ "import context_pandas1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Subfolder creation\n", "\n", "Subsequent notebooks are going to need a place to put raw and processed data.\n", "We create those folders in the cell below" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "lines_to_next_cell": 0, "scrolled": true }, "outputs": [], "source": [ "processed_dir = context_pandas1.data_dir / \"processed\"\n", "raw_dir = context_pandas1.data_dir / \"raw\"\n", "processed_dir.mkdir(parents=True, exist_ok=True)\n", "raw_dir.mkdir(parents=True, exist_ok=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pandas dataframes vs. numpy arrays\n", "\n", "* Dataframes are **column oriented**, arrays are **row oriented**\n", "* Array items are all of the same dtype (i.e. numpy.float32), dataframe columns can\n", " have different types (e.g.strings vs. integers)\n", "* Dataframe columns can be indexed by name (e.g. \"Total area of basin\") or by integer index\n", "* Dataframe rows can be indexed by number of by a special index (e.g. postal code)\n", "* Dataframe objects have dozens of methods to summarize and manipulate the data they hold, making\n", " them similar in features to a lightweight relational database." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Intro to Pandas\n", "\n", "- `pandas` = [Python Data Analysis Library](https://pandas.pydata.org/)\n", "- Best book: [Python for data analysis](https://github.com/wesm/pydata-book) by Wes McKinney\n", "- Jennifer Walker's [Pandas cheatsheet](pandas-cheatsheet.ipynb)\n", "- Library for working with **labelled** tabular data (1-D and 2-D)\n", " - Data formats include: comma separated values (CSV) and other text files, Excel spreadsheets, HDF5, [and others](https://pandas.pydata.org/pandas-docs/stable/io.html)\n", "- With `pandas` you can do pretty much everything you would in a spreadsheet, plus a whole lot more!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> If you're working with higher dimensional data and/or netCDF files, check out the excellent [xarray library](http://xarray.pydata.org/en/stable/), which brings the labelled data power of `pandas` to N-dimensional arrays" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why Pandas?\n", "- Working with large data files and complex calculations\n", "- Dealing with messy and missing data\n", "- Merging data from multiple files\n", "- Timeseries analysis\n", "- Automate repetitive tasks\n", "- Combine with other Python libraries to create beautiful and fully customized visualizations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading a CSV file\n", "\n", "We'll be working with the file `weather_YVR.csv` in the `data` sub-folder.\n", "- Environment Canada daily weather measurements at Vancouver Airport from 1938-2017." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's read the CSV file into our notebook with the function `read_csv` from the `pandas` library.\n", "- To access functions in the library, we use dot notation again: `pandas.read_csv()`\n", "- Our input to the read_csv function is the file path as a string: `'data/weather_YVR.csv'`\n", "\n", "We'll store the data as a dataframe called `weather`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The pathlib module\n", "\n", "The cell below constructs a [Path object](https://realpython.com/python-pathlib/). Note the\n", "direction of the \"/\" separator. This would not be the way that we would\n", "specify a file on windows (that would be 'data\\weather_YVR.csv' -- Path objects\n", "hide this complexity by understanding whether we are working on windows, linux or macos and\n", "just doing the right thing." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "weather_file = context_pandas1.data_dir / \"weather_YVR.csv\"" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "weather = pandas.read_csv(weather_file)" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "> Pro Tips!\n", "- Try typing `pandas.re` and then press Tab and select `read_csv` from the auto-complete options\n", "- Auto-complete even works for file paths inside a string!" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateYearMonthDayT_mean (C)T_high (C)T_low (C)Rain (mm)Snow (cm)Total Precip (mm)
01938-01-011938114.49.4-0.6NaNNaN0.3
11938-01-021938124.57.21.7NaNNaN0.5
21938-01-031938131.77.2-3.90.00.00.0
31938-01-041938142.27.2-2.80.00.00.0
41938-01-051938152.27.2-2.80.00.00.0
51938-01-061938162.35.6-1.10.00.00.0
61938-01-071938172.24.40.00.00.00.0
71938-01-081938182.56.7-1.7NaNNaN0.0
81938-01-091938194.25.62.8NaNNaN4.3
91938-01-1019381105.87.24.4NaNNaN3.6
101938-01-1119381114.26.71.7NaNNaN0.0
111938-01-1219381127.810.65.0NaNNaN5.3
121938-01-1319381136.18.33.9NaNNaN28.7
131938-01-1419381146.78.35.0NaNNaN18.3
141938-01-1519381154.58.30.60.00.00.0
151938-01-1619381164.56.72.2NaNNaN6.6
161938-01-1719381172.87.2-1.70.00.00.0
171938-01-1819381184.87.81.7NaNNaN5.8
181938-01-1919381193.46.70.00.00.00.0
191938-01-2019381202.86.1-0.6NaNNaN8.9
201938-01-2119381215.08.31.7NaNNaN2.5
211938-01-2219381223.18.3-2.2NaNNaN3.8
221938-01-2319381232.87.8-2.20.00.00.0
231938-01-2419381243.46.70.0NaNNaN0.0
241938-01-2519381254.87.81.70.00.00.0
251938-01-2619381260.33.9-3.30.00.00.0
261938-01-2719381272.86.1-0.60.00.00.0
271938-01-2819381284.56.72.2NaNNaN8.9
281938-01-2919381291.77.8-4.40.00.00.0
291938-01-301938130-2.31.1-5.6NaNNaN5.6
.................................
291602017-12-0220171225.36.83.75.20.05.2
291612017-12-0320171233.66.01.10.00.00.0
291622017-12-0420171243.55.81.10.00.00.0
291632017-12-0520171252.66.3-1.20.00.00.0
291642017-12-0620171262.25.8-1.50.00.00.0
291652017-12-0720171272.94.21.50.00.00.0
291662017-12-0820171281.93.50.20.00.00.0
291672017-12-0920171291.94.5-0.80.00.00.0
291682017-12-10201712104.17.70.50.00.00.0
291692017-12-11201712113.88.9-1.30.00.00.0
291702017-12-12201712124.87.61.90.00.00.0
291712017-12-13201712134.47.90.80.00.00.0
291722017-12-14201712143.06.5-0.50.00.00.0
291732017-12-15201712154.26.91.40.80.00.8
291742017-12-16201712164.16.51.65.80.05.8
291752017-12-17201712176.07.44.640.60.040.6
291762017-12-18201712186.27.35.03.00.03.0
291772017-12-19201712192.85.40.142.61.044.2
291782017-12-20201712201.75.0-1.70.00.00.0
291792017-12-2120171221-0.22.3-2.70.00.00.0
291802017-12-22201712221.55.5-2.60.00.00.0
291812017-12-2320171223-1.52.8-5.80.00.00.0
291822017-12-2420171224-2.00.9-4.90.00.00.0
291832017-12-2520171225-0.31.0-1.50.00.00.0
291842017-12-2620171226-0.40.7-1.40.00.00.0
291852017-12-27201712270.01.9-1.93.00.23.2
291862017-12-28201712283.45.71.120.80.020.8
291872017-12-29201712292.84.70.927.60.027.6
291882017-12-30201712301.34.3-1.82.20.02.2
291892017-12-3120171231-0.13.8-3.90.00.00.0
\n", "

29190 rows × 10 columns

\n", "
" ], "text/plain": [ " Date Year Month Day T_mean (C) T_high (C) T_low (C) \\\n", "0 1938-01-01 1938 1 1 4.4 9.4 -0.6 \n", "1 1938-01-02 1938 1 2 4.5 7.2 1.7 \n", "2 1938-01-03 1938 1 3 1.7 7.2 -3.9 \n", "3 1938-01-04 1938 1 4 2.2 7.2 -2.8 \n", "4 1938-01-05 1938 1 5 2.2 7.2 -2.8 \n", "5 1938-01-06 1938 1 6 2.3 5.6 -1.1 \n", "6 1938-01-07 1938 1 7 2.2 4.4 0.0 \n", "7 1938-01-08 1938 1 8 2.5 6.7 -1.7 \n", "8 1938-01-09 1938 1 9 4.2 5.6 2.8 \n", "9 1938-01-10 1938 1 10 5.8 7.2 4.4 \n", "10 1938-01-11 1938 1 11 4.2 6.7 1.7 \n", "11 1938-01-12 1938 1 12 7.8 10.6 5.0 \n", "12 1938-01-13 1938 1 13 6.1 8.3 3.9 \n", "13 1938-01-14 1938 1 14 6.7 8.3 5.0 \n", "14 1938-01-15 1938 1 15 4.5 8.3 0.6 \n", "15 1938-01-16 1938 1 16 4.5 6.7 2.2 \n", "16 1938-01-17 1938 1 17 2.8 7.2 -1.7 \n", "17 1938-01-18 1938 1 18 4.8 7.8 1.7 \n", "18 1938-01-19 1938 1 19 3.4 6.7 0.0 \n", "19 1938-01-20 1938 1 20 2.8 6.1 -0.6 \n", "20 1938-01-21 1938 1 21 5.0 8.3 1.7 \n", "21 1938-01-22 1938 1 22 3.1 8.3 -2.2 \n", "22 1938-01-23 1938 1 23 2.8 7.8 -2.2 \n", "23 1938-01-24 1938 1 24 3.4 6.7 0.0 \n", "24 1938-01-25 1938 1 25 4.8 7.8 1.7 \n", "25 1938-01-26 1938 1 26 0.3 3.9 -3.3 \n", "26 1938-01-27 1938 1 27 2.8 6.1 -0.6 \n", "27 1938-01-28 1938 1 28 4.5 6.7 2.2 \n", "28 1938-01-29 1938 1 29 1.7 7.8 -4.4 \n", "29 1938-01-30 1938 1 30 -2.3 1.1 -5.6 \n", "... ... ... ... ... ... ... ... \n", "29160 2017-12-02 2017 12 2 5.3 6.8 3.7 \n", "29161 2017-12-03 2017 12 3 3.6 6.0 1.1 \n", "29162 2017-12-04 2017 12 4 3.5 5.8 1.1 \n", "29163 2017-12-05 2017 12 5 2.6 6.3 -1.2 \n", "29164 2017-12-06 2017 12 6 2.2 5.8 -1.5 \n", "29165 2017-12-07 2017 12 7 2.9 4.2 1.5 \n", "29166 2017-12-08 2017 12 8 1.9 3.5 0.2 \n", "29167 2017-12-09 2017 12 9 1.9 4.5 -0.8 \n", "29168 2017-12-10 2017 12 10 4.1 7.7 0.5 \n", "29169 2017-12-11 2017 12 11 3.8 8.9 -1.3 \n", "29170 2017-12-12 2017 12 12 4.8 7.6 1.9 \n", "29171 2017-12-13 2017 12 13 4.4 7.9 0.8 \n", "29172 2017-12-14 2017 12 14 3.0 6.5 -0.5 \n", "29173 2017-12-15 2017 12 15 4.2 6.9 1.4 \n", "29174 2017-12-16 2017 12 16 4.1 6.5 1.6 \n", "29175 2017-12-17 2017 12 17 6.0 7.4 4.6 \n", "29176 2017-12-18 2017 12 18 6.2 7.3 5.0 \n", "29177 2017-12-19 2017 12 19 2.8 5.4 0.1 \n", "29178 2017-12-20 2017 12 20 1.7 5.0 -1.7 \n", "29179 2017-12-21 2017 12 21 -0.2 2.3 -2.7 \n", "29180 2017-12-22 2017 12 22 1.5 5.5 -2.6 \n", "29181 2017-12-23 2017 12 23 -1.5 2.8 -5.8 \n", "29182 2017-12-24 2017 12 24 -2.0 0.9 -4.9 \n", "29183 2017-12-25 2017 12 25 -0.3 1.0 -1.5 \n", "29184 2017-12-26 2017 12 26 -0.4 0.7 -1.4 \n", "29185 2017-12-27 2017 12 27 0.0 1.9 -1.9 \n", "29186 2017-12-28 2017 12 28 3.4 5.7 1.1 \n", "29187 2017-12-29 2017 12 29 2.8 4.7 0.9 \n", "29188 2017-12-30 2017 12 30 1.3 4.3 -1.8 \n", "29189 2017-12-31 2017 12 31 -0.1 3.8 -3.9 \n", "\n", " Rain (mm) Snow (cm) Total Precip (mm) \n", "0 NaN NaN 0.3 \n", "1 NaN NaN 0.5 \n", "2 0.0 0.0 0.0 \n", "3 0.0 0.0 0.0 \n", "4 0.0 0.0 0.0 \n", "5 0.0 0.0 0.0 \n", "6 0.0 0.0 0.0 \n", "7 NaN NaN 0.0 \n", "8 NaN NaN 4.3 \n", "9 NaN NaN 3.6 \n", "10 NaN NaN 0.0 \n", "11 NaN NaN 5.3 \n", "12 NaN NaN 28.7 \n", "13 NaN NaN 18.3 \n", "14 0.0 0.0 0.0 \n", "15 NaN NaN 6.6 \n", "16 0.0 0.0 0.0 \n", "17 NaN NaN 5.8 \n", "18 0.0 0.0 0.0 \n", "19 NaN NaN 8.9 \n", "20 NaN NaN 2.5 \n", "21 NaN NaN 3.8 \n", "22 0.0 0.0 0.0 \n", "23 NaN NaN 0.0 \n", "24 0.0 0.0 0.0 \n", "25 0.0 0.0 0.0 \n", "26 0.0 0.0 0.0 \n", "27 NaN NaN 8.9 \n", "28 0.0 0.0 0.0 \n", "29 NaN NaN 5.6 \n", "... ... ... ... \n", "29160 5.2 0.0 5.2 \n", "29161 0.0 0.0 0.0 \n", "29162 0.0 0.0 0.0 \n", "29163 0.0 0.0 0.0 \n", "29164 0.0 0.0 0.0 \n", "29165 0.0 0.0 0.0 \n", "29166 0.0 0.0 0.0 \n", "29167 0.0 0.0 0.0 \n", "29168 0.0 0.0 0.0 \n", "29169 0.0 0.0 0.0 \n", "29170 0.0 0.0 0.0 \n", "29171 0.0 0.0 0.0 \n", "29172 0.0 0.0 0.0 \n", "29173 0.8 0.0 0.8 \n", "29174 5.8 0.0 5.8 \n", "29175 40.6 0.0 40.6 \n", "29176 3.0 0.0 3.0 \n", "29177 42.6 1.0 44.2 \n", "29178 0.0 0.0 0.0 \n", "29179 0.0 0.0 0.0 \n", "29180 0.0 0.0 0.0 \n", "29181 0.0 0.0 0.0 \n", "29182 0.0 0.0 0.0 \n", "29183 0.0 0.0 0.0 \n", "29184 0.0 0.0 0.0 \n", "29185 3.0 0.2 3.2 \n", "29186 20.8 0.0 20.8 \n", "29187 27.6 0.0 27.6 \n", "29188 2.2 0.0 2.2 \n", "29189 0.0 0.0 0.0 \n", "\n", "[29190 rows x 10 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Only the first 30 and last 30 rows are displayed (but the data is all there in our `weather` variable)\n", "- You may notice some weird `NaN` values—these represent missing data (`NaN` = \"not a number\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "What type of object is `weather`?" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "pandas.core.frame.DataFrame" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(weather)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- `weather` is a **DataFrame**, a data structure from the `pandas` library\n", " - A DataFrame is a 2-dimensional array (organized into rows and columns, like a table in a spreadsheet)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- When we display `weather`, the integer numbers in bold on the left are the DataFrame's **index**\n", " - In this case, the index is simply a range of integers corresponding with the row numbers" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateYearMonthDayT_mean (C)T_high (C)T_low (C)Rain (mm)Snow (cm)Total Precip (mm)
01938-01-011938114.49.4-0.6NaNNaN0.3
11938-01-021938124.57.21.7NaNNaN0.5
21938-01-031938131.77.2-3.90.00.00.0
31938-01-041938142.27.2-2.80.00.00.0
41938-01-051938152.27.2-2.80.00.00.0
51938-01-061938162.35.6-1.10.00.00.0
61938-01-071938172.24.40.00.00.00.0
71938-01-081938182.56.7-1.7NaNNaN0.0
81938-01-091938194.25.62.8NaNNaN4.3
91938-01-1019381105.87.24.4NaNNaN3.6
101938-01-1119381114.26.71.7NaNNaN0.0
111938-01-1219381127.810.65.0NaNNaN5.3
121938-01-1319381136.18.33.9NaNNaN28.7
131938-01-1419381146.78.35.0NaNNaN18.3
141938-01-1519381154.58.30.60.00.00.0
151938-01-1619381164.56.72.2NaNNaN6.6
161938-01-1719381172.87.2-1.70.00.00.0
171938-01-1819381184.87.81.7NaNNaN5.8
181938-01-1919381193.46.70.00.00.00.0
191938-01-2019381202.86.1-0.6NaNNaN8.9
201938-01-2119381215.08.31.7NaNNaN2.5
211938-01-2219381223.18.3-2.2NaNNaN3.8
221938-01-2319381232.87.8-2.20.00.00.0
231938-01-2419381243.46.70.0NaNNaN0.0
241938-01-2519381254.87.81.70.00.00.0
251938-01-2619381260.33.9-3.30.00.00.0
261938-01-2719381272.86.1-0.60.00.00.0
271938-01-2819381284.56.72.2NaNNaN8.9
281938-01-2919381291.77.8-4.40.00.00.0
291938-01-301938130-2.31.1-5.6NaNNaN5.6
.................................
291602017-12-0220171225.36.83.75.20.05.2
291612017-12-0320171233.66.01.10.00.00.0
291622017-12-0420171243.55.81.10.00.00.0
291632017-12-0520171252.66.3-1.20.00.00.0
291642017-12-0620171262.25.8-1.50.00.00.0
291652017-12-0720171272.94.21.50.00.00.0
291662017-12-0820171281.93.50.20.00.00.0
291672017-12-0920171291.94.5-0.80.00.00.0
291682017-12-10201712104.17.70.50.00.00.0
291692017-12-11201712113.88.9-1.30.00.00.0
291702017-12-12201712124.87.61.90.00.00.0
291712017-12-13201712134.47.90.80.00.00.0
291722017-12-14201712143.06.5-0.50.00.00.0
291732017-12-15201712154.26.91.40.80.00.8
291742017-12-16201712164.16.51.65.80.05.8
291752017-12-17201712176.07.44.640.60.040.6
291762017-12-18201712186.27.35.03.00.03.0
291772017-12-19201712192.85.40.142.61.044.2
291782017-12-20201712201.75.0-1.70.00.00.0
291792017-12-2120171221-0.22.3-2.70.00.00.0
291802017-12-22201712221.55.5-2.60.00.00.0
291812017-12-2320171223-1.52.8-5.80.00.00.0
291822017-12-2420171224-2.00.9-4.90.00.00.0
291832017-12-2520171225-0.31.0-1.50.00.00.0
291842017-12-2620171226-0.40.7-1.40.00.00.0
291852017-12-27201712270.01.9-1.93.00.23.2
291862017-12-28201712283.45.71.120.80.020.8
291872017-12-29201712292.84.70.927.60.027.6
291882017-12-30201712301.34.3-1.82.20.02.2
291892017-12-3120171231-0.13.8-3.90.00.00.0
\n", "

29190 rows × 10 columns

\n", "
" ], "text/plain": [ " Date Year Month Day T_mean (C) T_high (C) T_low (C) \\\n", "0 1938-01-01 1938 1 1 4.4 9.4 -0.6 \n", "1 1938-01-02 1938 1 2 4.5 7.2 1.7 \n", "2 1938-01-03 1938 1 3 1.7 7.2 -3.9 \n", "3 1938-01-04 1938 1 4 2.2 7.2 -2.8 \n", "4 1938-01-05 1938 1 5 2.2 7.2 -2.8 \n", "5 1938-01-06 1938 1 6 2.3 5.6 -1.1 \n", "6 1938-01-07 1938 1 7 2.2 4.4 0.0 \n", "7 1938-01-08 1938 1 8 2.5 6.7 -1.7 \n", "8 1938-01-09 1938 1 9 4.2 5.6 2.8 \n", "9 1938-01-10 1938 1 10 5.8 7.2 4.4 \n", "10 1938-01-11 1938 1 11 4.2 6.7 1.7 \n", "11 1938-01-12 1938 1 12 7.8 10.6 5.0 \n", "12 1938-01-13 1938 1 13 6.1 8.3 3.9 \n", "13 1938-01-14 1938 1 14 6.7 8.3 5.0 \n", "14 1938-01-15 1938 1 15 4.5 8.3 0.6 \n", "15 1938-01-16 1938 1 16 4.5 6.7 2.2 \n", "16 1938-01-17 1938 1 17 2.8 7.2 -1.7 \n", "17 1938-01-18 1938 1 18 4.8 7.8 1.7 \n", "18 1938-01-19 1938 1 19 3.4 6.7 0.0 \n", "19 1938-01-20 1938 1 20 2.8 6.1 -0.6 \n", "20 1938-01-21 1938 1 21 5.0 8.3 1.7 \n", "21 1938-01-22 1938 1 22 3.1 8.3 -2.2 \n", "22 1938-01-23 1938 1 23 2.8 7.8 -2.2 \n", "23 1938-01-24 1938 1 24 3.4 6.7 0.0 \n", "24 1938-01-25 1938 1 25 4.8 7.8 1.7 \n", "25 1938-01-26 1938 1 26 0.3 3.9 -3.3 \n", "26 1938-01-27 1938 1 27 2.8 6.1 -0.6 \n", "27 1938-01-28 1938 1 28 4.5 6.7 2.2 \n", "28 1938-01-29 1938 1 29 1.7 7.8 -4.4 \n", "29 1938-01-30 1938 1 30 -2.3 1.1 -5.6 \n", "... ... ... ... ... ... ... ... \n", "29160 2017-12-02 2017 12 2 5.3 6.8 3.7 \n", "29161 2017-12-03 2017 12 3 3.6 6.0 1.1 \n", "29162 2017-12-04 2017 12 4 3.5 5.8 1.1 \n", "29163 2017-12-05 2017 12 5 2.6 6.3 -1.2 \n", "29164 2017-12-06 2017 12 6 2.2 5.8 -1.5 \n", "29165 2017-12-07 2017 12 7 2.9 4.2 1.5 \n", "29166 2017-12-08 2017 12 8 1.9 3.5 0.2 \n", "29167 2017-12-09 2017 12 9 1.9 4.5 -0.8 \n", "29168 2017-12-10 2017 12 10 4.1 7.7 0.5 \n", "29169 2017-12-11 2017 12 11 3.8 8.9 -1.3 \n", "29170 2017-12-12 2017 12 12 4.8 7.6 1.9 \n", "29171 2017-12-13 2017 12 13 4.4 7.9 0.8 \n", "29172 2017-12-14 2017 12 14 3.0 6.5 -0.5 \n", "29173 2017-12-15 2017 12 15 4.2 6.9 1.4 \n", "29174 2017-12-16 2017 12 16 4.1 6.5 1.6 \n", "29175 2017-12-17 2017 12 17 6.0 7.4 4.6 \n", "29176 2017-12-18 2017 12 18 6.2 7.3 5.0 \n", "29177 2017-12-19 2017 12 19 2.8 5.4 0.1 \n", "29178 2017-12-20 2017 12 20 1.7 5.0 -1.7 \n", "29179 2017-12-21 2017 12 21 -0.2 2.3 -2.7 \n", "29180 2017-12-22 2017 12 22 1.5 5.5 -2.6 \n", "29181 2017-12-23 2017 12 23 -1.5 2.8 -5.8 \n", "29182 2017-12-24 2017 12 24 -2.0 0.9 -4.9 \n", "29183 2017-12-25 2017 12 25 -0.3 1.0 -1.5 \n", "29184 2017-12-26 2017 12 26 -0.4 0.7 -1.4 \n", "29185 2017-12-27 2017 12 27 0.0 1.9 -1.9 \n", "29186 2017-12-28 2017 12 28 3.4 5.7 1.1 \n", "29187 2017-12-29 2017 12 29 2.8 4.7 0.9 \n", "29188 2017-12-30 2017 12 30 1.3 4.3 -1.8 \n", "29189 2017-12-31 2017 12 31 -0.1 3.8 -3.9 \n", "\n", " Rain (mm) Snow (cm) Total Precip (mm) \n", "0 NaN NaN 0.3 \n", "1 NaN NaN 0.5 \n", "2 0.0 0.0 0.0 \n", "3 0.0 0.0 0.0 \n", "4 0.0 0.0 0.0 \n", "5 0.0 0.0 0.0 \n", "6 0.0 0.0 0.0 \n", "7 NaN NaN 0.0 \n", "8 NaN NaN 4.3 \n", "9 NaN NaN 3.6 \n", "10 NaN NaN 0.0 \n", "11 NaN NaN 5.3 \n", "12 NaN NaN 28.7 \n", "13 NaN NaN 18.3 \n", "14 0.0 0.0 0.0 \n", "15 NaN NaN 6.6 \n", "16 0.0 0.0 0.0 \n", "17 NaN NaN 5.8 \n", "18 0.0 0.0 0.0 \n", "19 NaN NaN 8.9 \n", "20 NaN NaN 2.5 \n", "21 NaN NaN 3.8 \n", "22 0.0 0.0 0.0 \n", "23 NaN NaN 0.0 \n", "24 0.0 0.0 0.0 \n", "25 0.0 0.0 0.0 \n", "26 0.0 0.0 0.0 \n", "27 NaN NaN 8.9 \n", "28 0.0 0.0 0.0 \n", "29 NaN NaN 5.6 \n", "... ... ... ... \n", "29160 5.2 0.0 5.2 \n", "29161 0.0 0.0 0.0 \n", "29162 0.0 0.0 0.0 \n", "29163 0.0 0.0 0.0 \n", "29164 0.0 0.0 0.0 \n", "29165 0.0 0.0 0.0 \n", "29166 0.0 0.0 0.0 \n", "29167 0.0 0.0 0.0 \n", "29168 0.0 0.0 0.0 \n", "29169 0.0 0.0 0.0 \n", "29170 0.0 0.0 0.0 \n", "29171 0.0 0.0 0.0 \n", "29172 0.0 0.0 0.0 \n", "29173 0.8 0.0 0.8 \n", "29174 5.8 0.0 5.8 \n", "29175 40.6 0.0 40.6 \n", "29176 3.0 0.0 3.0 \n", "29177 42.6 1.0 44.2 \n", "29178 0.0 0.0 0.0 \n", "29179 0.0 0.0 0.0 \n", "29180 0.0 0.0 0.0 \n", "29181 0.0 0.0 0.0 \n", "29182 0.0 0.0 0.0 \n", "29183 0.0 0.0 0.0 \n", "29184 0.0 0.0 0.0 \n", "29185 3.0 0.2 3.2 \n", "29186 20.8 0.0 20.8 \n", "29187 27.6 0.0 27.6 \n", "29188 2.2 0.0 2.2 \n", "29189 0.0 0.0 0.0 \n", "\n", "[29190 rows x 10 columns]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For large DataFrames, it's often useful to display just the first few or last few rows:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateYearMonthDayT_mean (C)T_high (C)T_low (C)Rain (mm)Snow (cm)Total Precip (mm)
01938-01-011938114.49.4-0.6NaNNaN0.3
11938-01-021938124.57.21.7NaNNaN0.5
21938-01-031938131.77.2-3.90.00.00.0
31938-01-041938142.27.2-2.80.00.00.0
41938-01-051938152.27.2-2.80.00.00.0
\n", "
" ], "text/plain": [ " Date Year Month Day T_mean (C) T_high (C) T_low (C) Rain (mm) \\\n", "0 1938-01-01 1938 1 1 4.4 9.4 -0.6 NaN \n", "1 1938-01-02 1938 1 2 4.5 7.2 1.7 NaN \n", "2 1938-01-03 1938 1 3 1.7 7.2 -3.9 0.0 \n", "3 1938-01-04 1938 1 4 2.2 7.2 -2.8 0.0 \n", "4 1938-01-05 1938 1 5 2.2 7.2 -2.8 0.0 \n", "\n", " Snow (cm) Total Precip (mm) \n", "0 NaN 0.3 \n", "1 NaN 0.5 \n", "2 0.0 0.0 \n", "3 0.0 0.0 \n", "4 0.0 0.0 " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.head()" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "fragment" } }, "source": [ "The `head` method returns a new DataFrame consisting of the first `n` rows (default 5)\n", "\n", "\n", "> Pro Tips!\n", "> - To display the documentation for this method, you can run the command `weather.head?` in your Jupyter notebook\n", "> - To see other methods available for the DataFrame, type `weather.` followed by Tab for auto-complete options" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First two rows:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateYearMonthDayT_mean (C)T_high (C)T_low (C)Rain (mm)Snow (cm)Total Precip (mm)
01938-01-011938114.49.4-0.6NaNNaN0.3
11938-01-021938124.57.21.7NaNNaN0.5
\n", "
" ], "text/plain": [ " Date Year Month Day T_mean (C) T_high (C) T_low (C) Rain (mm) \\\n", "0 1938-01-01 1938 1 1 4.4 9.4 -0.6 NaN \n", "1 1938-01-02 1938 1 2 4.5 7.2 1.7 NaN \n", "\n", " Snow (cm) Total Precip (mm) \n", "0 NaN 0.3 \n", "1 NaN 0.5 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.head(2)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "# Last four rows:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
DateYearMonthDayT_mean (C)T_high (C)T_low (C)Rain (mm)Snow (cm)Total Precip (mm)
291862017-12-28201712283.45.71.120.80.020.8
291872017-12-29201712292.84.70.927.60.027.6
291882017-12-30201712301.34.3-1.82.20.02.2
291892017-12-3120171231-0.13.8-3.90.00.00.0
\n", "
" ], "text/plain": [ " Date Year Month Day T_mean (C) T_high (C) T_low (C) \\\n", "29186 2017-12-28 2017 12 28 3.4 5.7 1.1 \n", "29187 2017-12-29 2017 12 29 2.8 4.7 0.9 \n", "29188 2017-12-30 2017 12 30 1.3 4.3 -1.8 \n", "29189 2017-12-31 2017 12 31 -0.1 3.8 -3.9 \n", "\n", " Rain (mm) Snow (cm) Total Precip (mm) \n", "29186 20.8 0.0 20.8 \n", "29187 27.6 0.0 27.6 \n", "29188 2.2 0.0 2.2 \n", "29189 0.0 0.0 0.0 " ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.tail(4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data at a Glance\n", "\n", "`pandas` provides many ways to quickly and easily summarize your data:\n", " - How many rows and columns are there?\n", " - What are all the column names and what type of data is in each column?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Numerical data: What is the average and range of the values?\n", "- Text data: What are the unique values and how often does each occur?\n", "- How many values are missing in each column or row?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Number of rows and columns:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(29190, 10)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.shape" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "# - The DataFrame `weather` has 29190 rows and 10 columns\n", "# - The index does not count as a column\n", "# - Notice there are no parentheses at the end of `weather.shape`\n", "# - `shape` is a **data attribute** of the variable `weather`" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "tuple" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(weather.shape)" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "# The data in the `shape` attribute is stored as a **tuple**, which is similar to a list.\n", "#\n", "# - Items in a tuple are enclosed in `()` instead of `[]`\n", "# - Tuples are immutable - you can't modify individual items inside a tuple" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "- Within a column of a DataFrame, the data must all be of the same type\n", "- We can find out the names and data types of each column from the `dtypes` attribute:" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Date object\n", "Year int64\n", "Month int64\n", "Day int64\n", "T_mean (C) float64\n", "T_high (C) float64\n", "T_low (C) float64\n", "Rain (mm) float64\n", "Snow (cm) float64\n", "Total Precip (mm) float64\n", "dtype: object" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In a `pandas` DataFrame, a column containing text data (or containing a mix of text and numbers) is assigned a `dtype` of `object` and is treated as a column of strings.\n", "\n", "`int64` and `float64` are integer and float, respectively.\n", "- The 64 at the end means that they are stored as 64-bit numbers in memory\n", "- These data types are equivalent to `int` and `float` in Python (`pandas` is a just a bit more explicit in how it names them)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we just want a list of the column names, we can use the `columns` attribute:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "scrolled": true, "slideshow": { "slide_type": "-" } }, "outputs": [ { "data": { "text/plain": [ "Index(['Date', 'Year', 'Month', 'Day', 'T_mean (C)', 'T_high (C)', 'T_low (C)',\n", " 'Rain (mm)', 'Snow (cm)', 'Total Precip (mm)'],\n", " dtype='object')" ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Simple Summary Statistics\n", "\n", "The `describe` method computes simple summary statistics and returns them as a DataFrame:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
YearMonthDayT_mean (C)T_high (C)T_low (C)Rain (mm)Snow (cm)Total Precip (mm)
count29190.00000029190.00000029190.00000029167.00000029169.00000029180.00000029126.00000029138.00000029166.000000
mean1977.4797536.52233615.72922210.13369213.7807676.4400622.9857310.1217793.107351
std23.0938913.4485848.8011675.8035496.4088155.5141296.2311141.1462986.325946
min1938.0000001.0000001.000000-14.500000-11.100000-17.8000000.0000000.0000000.000000
25%1957.0000004.0000008.0000005.9000008.9000002.6000000.0000000.0000000.000000
50%1977.0000007.00000016.00000010.00000013.3000006.7000000.0000000.0000000.000000
75%1997.75000010.00000023.00000015.00000018.90000011.1000003.2000000.0000003.400000
max2017.00000012.00000031.00000028.40000034.40000022.40000091.60000041.00000091.600000
\n", "
" ], "text/plain": [ " Year Month Day T_mean (C) T_high (C) \\\n", "count 29190.000000 29190.000000 29190.000000 29167.000000 29169.000000 \n", "mean 1977.479753 6.522336 15.729222 10.133692 13.780767 \n", "std 23.093891 3.448584 8.801167 5.803549 6.408815 \n", "min 1938.000000 1.000000 1.000000 -14.500000 -11.100000 \n", "25% 1957.000000 4.000000 8.000000 5.900000 8.900000 \n", "50% 1977.000000 7.000000 16.000000 10.000000 13.300000 \n", "75% 1997.750000 10.000000 23.000000 15.000000 18.900000 \n", "max 2017.000000 12.000000 31.000000 28.400000 34.400000 \n", "\n", " T_low (C) Rain (mm) Snow (cm) Total Precip (mm) \n", "count 29180.000000 29126.000000 29138.000000 29166.000000 \n", "mean 6.440062 2.985731 0.121779 3.107351 \n", "std 5.514129 6.231114 1.146298 6.325946 \n", "min -17.800000 0.000000 0.000000 0.000000 \n", "25% 2.600000 0.000000 0.000000 0.000000 \n", "50% 6.700000 0.000000 0.000000 0.000000 \n", "75% 11.100000 3.200000 0.000000 3.400000 \n", "max 22.400000 91.600000 41.000000 91.600000 " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `describe` method is a way to quickly summarize the averages, extremes, and variability of each numerical data column." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can look at each statistic individually with methods such as `mean`, `median`, `min`, `max`,`std`, and `count`" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Year 1977.479753\n", "Month 6.522336\n", "Day 15.729222\n", "T_mean (C) 10.133692\n", "T_high (C) 13.780767\n", "T_low (C) 6.440062\n", "Rain (mm) 2.985731\n", "Snow (cm) 0.121779\n", "Total Precip (mm) 3.107351\n", "dtype: float64" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "weather.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise\n", "\n", "For this exercise, we will explore data about countries around the world, combined from multiple sources by the [Gapminder foundation](https://www.gapminder.org/about-gapminder/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Gapminder is an independent Swedish foundation that fights devastating misconceptions about global development and promotes a fact-based world view through the production of free teaching and data exploration resources." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Data Overview\n", "\n", "The columns of `data/gapminder_world_data_2018.csv` are:\n", "\n", "| Column | Description |\n", "|-----------------------|------------------------------------|\n", "| country | Country name |\n", "| population | Population in the country |\n", "| region | Continent the country belongs to |\n", "| sub_region | Sub regions as defined by |\n", "| income_group | Income group [as specified by the world bank](https://datahelpdesk.worldbank.org/knowledgebase/articles/378833-how-are-the-income-group-thresholds-determined) |\n", "| life_expectancy | The average number of years a newborn child would
live if mortality patterns were to stay the same |\n", "| gdp_per_capita | GDP per capita (in USD) adjusted
for differences in purchasing power|\n", "| children_per_woman | Number of children born to each woman|\n", "| child_mortality | Deaths of children under 5 years
of age per 1000 live births|\n", "| pop_density | Average number of people per km$^2$|" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> You may want to refer to the [Pandas cheatsheet](pandas-cheatsheet.ipynb) as you work through the exercises.\n", "\n", "**a)** Read the file `context_pandas1.data_dir / \"gapminder_world_data_2018.csv\"` into a new DataFrame `world` and display the first 10 rows." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**b)** How many rows and columns does `world` have?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**c)** Display the names and data types of each column." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**d)** Display summary statistics with the `describe` method. What are the lowest and highest populations? How about lowest/highest population densities? Any guesses which countries these might be? (We'll find out the answers in Lesson 4!)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Bonus exercises\n", "\n", "**e) Data wrangling - dealing with header rows**\n", "\n", "The file `raw_dir / \"weather_YVR_1938.csv\"` contains the daily weather data for 1938, in the original format downloaded from Environment Canada. Open this file in cocalc.\n", "\n", "> Note that a CSV viewer wouldn't able to parse the data correctly because of the extra header rows at the beginning.\n", "\n", "- Now try reading the file into your notebook with `pandas.read_csv` and see what happens.\n", "\n", "If you look at the documentation for `pandas.read_csv`, you'll see a `skiprows` input buried amongst a few dozen other inputs for this function. This input tells `read_csv` how many rows to skip at the beginning of the file.\n", "- Try reading `data/raw/weather_YVR_1938.csv` again, but this time using a value of `24` for the `skiprows` keyword argument, and display the first 5 rows of the resulting DataFrame." ] } ], "metadata": { "jupytext": { "cell_metadata_filter": "all", "notebook_metadata_filter": "all", "text_representation": { "extension": ".py", "format_name": "percent", "format_version": "1.2", "jupytext_version": "1.0.2" } }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.7" }, "nbsphinx": { "execute": "never" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": {}, "toc_section_display": true, "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 2 }