Table of Contents
1 Download Environment Canada Daily Data
1.1 Set the context for this notebook
1.2 Station Inventory
1.3 Info for Selected Station
1.3.1 Download Data
Download Environment Canada Daily Data¶
[1]:
from pathlib import Path
import pandas as pd
import requests
Set the context for this notebook¶
Importing the context module will check to see whether data/processed and data/raw exist and complain if it can’t find them
[2]:
import context
in context.py, setting root_dir to /Users/phil/repos/eosc213_students/notebooks/pandas
******************************
context imported. Front of path:
/Users/phil/repos/eosc213_students/notebooks/pandas
/Users/phil/mini37/lib/python36.zip
******************************
Station Inventory¶
Instructions: copy and paste this url into a browser: (ftp://client_climate@ftp.tor.ec.gc.ca/Pub/Get_More_Data_Plus_de_donnees/)
To get the station inventory (1.3 Mbyte csv file), copy and paste this url into a browser (
%20is the blank space character)ftp://client_climate@ftp.tor.ec.gc.ca/Pub/Get_More_Data_Plus_de_donnees/Station%20Inventory%20EN.csv
We have saved a copy in
data/Station Inventory EN .csv
[3]:
#
# note that whitespace (blanks, tabs, newlines) in a tuple are discarded
# so we can split a long string up like this:
#
f = (
"ftp://client_climate@ftp.tor.ec.gc.ca/Pub/"
"Get_More_Data_Plus_de_donnees/Station%20Inventory%20EN.csv"
)
[4]:
inventory_file = context.data_dir / "Station Inventory EN.csv"
inventory = pd.read_csv(inventory_file, skiprows=3)
# Rename some of the columns to more convenient labels
cols_dict = {
"TC ID": "Airport Code",
"Station ID": "Env Canada ID",
"Latitude (Decimal Degrees)": "Latitude (deg)",
"Longitude (Decimal Degrees)": "Longitude (deg)",
}
inventory = inventory.rename(columns=cols_dict)
print(inventory.shape)
inventory.head()
(8756, 19)
[4]:
| Name | Province | Climate ID | Env Canada ID | WMO ID | Airport Code | Latitude (deg) | Longitude (deg) | Latitude | Longitude | Elevation (m) | First Year | Last Year | HLY First Year | HLY Last Year | DLY First Year | DLY Last Year | MLY First Year | MLY Last Year | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | ACTIVE PASS | BRITISH COLUMBIA | 1010066 | 14 | NaN | NaN | 48.87 | -123.28 | 485200000 | -1231700000 | 4.0 | 1984 | 1996 | NaN | NaN | 1984.0 | 1996.0 | 1984.0 | 1996.0 |
| 1 | ALBERT HEAD | BRITISH COLUMBIA | 1010235 | 15 | NaN | NaN | 48.40 | -123.48 | 482400000 | -1232900000 | 17.0 | 1971 | 1995 | NaN | NaN | 1971.0 | 1995.0 | 1971.0 | 1995.0 |
| 2 | BAMBERTON OCEAN CEMENT | BRITISH COLUMBIA | 1010595 | 16 | NaN | NaN | 48.58 | -123.52 | 483500000 | -1233100000 | 85.3 | 1961 | 1980 | NaN | NaN | 1961.0 | 1980.0 | 1961.0 | 1980.0 |
| 3 | BEAR CREEK | BRITISH COLUMBIA | 1010720 | 17 | NaN | NaN | 48.50 | -124.00 | 483000000 | -1240000000 | 350.5 | 1910 | 1971 | NaN | NaN | 1910.0 | 1971.0 | 1910.0 | 1971.0 |
| 4 | BEAVER LAKE | BRITISH COLUMBIA | 1010774 | 18 | NaN | NaN | 48.50 | -123.35 | 483000000 | -1232100000 | 61.0 | 1894 | 1952 | NaN | NaN | 1894.0 | 1952.0 | 1894.0 | 1952.0 |
Info for Selected Station¶
To download data for Vancouver Airport station (airport code YVR), we need the ID codes used by Environment Canada for this station. Here is how we find the numerical code ‘YVR’. Note that it has changed at some point from 889 to 51442
[5]:
station = "YVR"
# Extract the inventory row(s) corresponding to this station
station_info = inventory[inventory["Airport Code"] == station]
station_info
[5]:
| Name | Province | Climate ID | Env Canada ID | WMO ID | Airport Code | Latitude (deg) | Longitude (deg) | Latitude | Longitude | Elevation (m) | First Year | Last Year | HLY First Year | HLY Last Year | DLY First Year | DLY Last Year | MLY First Year | MLY Last Year | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 996 | VANCOUVER INTL A | BRITISH COLUMBIA | 1108395 | 51442 | 71892.0 | YVR | 49.19 | -123.18 | 491141000 | -1231102000 | 4.3 | 2013 | 2019 | 2013.0 | 2019.0 | 2013.0 | 2019.0 | NaN | NaN |
| 1008 | VANCOUVER INT'L A | BRITISH COLUMBIA | 1108447 | 889 | NaN | YVR | 49.20 | -123.18 | 491142000 | -1231055000 | 4.3 | 1937 | 2013 | 1953.0 | 2013.0 | 1937.0 | 2013.0 | 1937.0 | 2013.0 |
Download Data¶
First, define a function to download the CSV data using the Environment Canada API:
[6]:
def download_daily_raw(env_canada_id, year, savefile="test.csv", verbose=True):
"""Download CSV file of daily data for selected station and year"""
# URL endpoint and query parameters
url_endpoint = "http://climate.weather.gc.ca/climate_data/bulk_data_e.html"
params = {
"format": "csv",
"stationID": env_canada_id,
"Year": year,
"Month": "01",
"Day": "01",
"timeframe": "2",
"submit": " Download Data",
}
# Send GET request
response = requests.get(url_endpoint, params=params)
# Download CSV file
if verbose:
print(f"Saving to {savefile}")
with open(savefile, "wb") as f:
f.write(response.content)
return None
Note: The code below usesf-stringsto substitute variable values into a string
[7]:
# Early data (1937 to mid 2013)
stn_id_early = 889 # station id for YVR airport
years_early = range(1937, 2014)
for year in years_early:
savefile = context.raw_dir / Path(f"weather_daily_{station}_{stn_id_early}_{year}.csv")
download_daily_raw(stn_id_early, year, savefile=savefile)
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1937.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1938.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1939.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1940.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1941.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1942.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1943.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1944.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1945.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1946.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1947.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1948.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1949.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1950.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1951.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1952.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1953.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1954.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1955.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1956.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1957.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1958.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1959.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1960.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1961.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1962.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1963.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1964.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1965.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1966.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1967.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1968.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1969.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1970.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1971.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1972.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1973.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1974.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1975.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1976.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1977.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1978.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1979.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1980.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1981.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1982.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1983.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1984.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1985.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1986.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1987.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1988.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1989.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1990.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1991.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1992.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1993.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1994.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1995.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1996.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1997.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1998.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_1999.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2000.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2001.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2002.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2003.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2004.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2005.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2006.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2007.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2008.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2009.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2010.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2011.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2012.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_889_2013.csv
[8]:
# Recent data (mid 2013 to 2019)
stn_id_recent = 51442
years_recent = range(2013, 2020)
for year in years_recent:
savefile = (context.raw_dir /
Path(f"weather_daily_{station}_{stn_id_recent}_{year}.csv"))
download_daily_raw(stn_id_recent, year, savefile=savefile)
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_51442_2013.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_51442_2014.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_51442_2015.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_51442_2016.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_51442_2017.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_51442_2018.csv
Saving to /Users/phil/repos/eosc213_students/notebooks/pandas/data/raw/weather_daily_YVR_51442_2019.csv