TDM 40100: Project 01 - Intro to ML - Using Anvil

Let’s start out by starting a new Anvil session. If you do not remember how to do this, please read through Project 1 at the introduction TDM 10100 level.

Once you have started a new Anvil session, download the project template and upload it. Then, open this template in Jupyter notebook. Save it as a new file with the following naming convention: lastname_firstname_project#.ipynb. For example, doe_jane_project1.ipynb.

To make sure everything is working, run the following code cell:

Your output should be Hello, world!. If you see this, you are ready to move on to the next question.

Although question 1 is trivially easy, we still want you to (please) get into the habit of commenting on the work in each question. So (please) it would be helpful to write (in a separate cell) something like, "We are reminding ourselves how to use Anvil and how to print a line of output."

Now that we have our Jupyter Lab notebook set up, let’s begin working with the pandas library.

Pandas is a Python library that allows us to work with datasets in tabular form. There are functions for loading datasets, manipulating data, etc.

To start out with, let’s load the Iris dataset that is located at /anvil/projects/tdm/data/iris/Iris.csv.

To do this, you will need to import the pandas library and use the read_csv function to load the dataset.

Run the following code cell to load the dataset:

Now that our dataset is loaded, let’s take a look at the first 5 rows of the dataset. To do this, run the following code cell:

An important aspect of our dataframe for machine learning is the shape (rows, columns). As you will learn later, the shape will help us determine what kind of machine learning model will be the best fit, as well as how complex it may be.

To get the shape of the dataframe, run the following code cell:

This returns a tuple in the form (rows, columns).

Now that we have loaded the dataset, let’s investigate how we can manipulate the data.

One common operation is to select a subset of the data. This is done using the iloc function, which allows us to index the dataframe by row and column numbers.

To select the first n rows of the dataframe, we can use the iloc function with a slice: myDF.iloc[:n].

Write code to select the first 10 rows of the dataframe from Question 3 into a new dataframe called myDF_subset. Print the shape of myDF_subset to verify that you have selected the correct number of rows.

We can also use the iloc function to select specific columns. To select specific columns, we can also use a slice, however we must specify the rows we want first. To select all rows, we simply pass a colon :. For example, to select the first 10 rows and the first 3 columns, we could use the following code: myDF.iloc[:10, :3].

Write code to select the 40th through 50th rows (inclusive) and the 2nd and 4th columns of the dataframe from Question 3 into a new dataframe called myDF_subset2. Print the shape of myDF_subset2 to verify that you have selected the correct number of rows and columns.

The iloc function can also be used to filter rows based on a condition. For example, if we wanted all rows where the PetalWidthCm is greater than 1.5, we could use the following code: myDF.loc[myDF['PetalWidthCm'] > 1.5, :].

Write code to select all rows where SepalLengthCm is less than 5.0 into a new dataframe called myDF_subset3. How many rows are in this dataframe?

Another common operation is to remove column(s) from the dataframe. This is done using the drop function.

The most readable way to drop a column is by dropping it by name. To drop column(s) by name, you can use the following syntax: myDF.drop(['column1_name', 'column2_name', …​], axis=1). The axis=1 argument tells pandas to drop columns, not rows.

Write code to drop the Id column from the myDF_subset into a new dataframe called myDF_without_id. Print the shape of the dataframe to verify that the column has been removed.

Additionally, we can extract columns from a dataframe into a new dataframe. Extracting a column is very simple: myDF['column_name'] will return a pandas series containing the values of the column. To extract multiple columns, you can pass a list of column names: myDF[['column1_name', 'column2_name', …​]]. To then store these series into a new dataframe, we can simply cast the series into a dataframe: pd.DataFrame(myDF['column_name']).

Write code to extract the Species and SepalWidthCm columns from the myDF_without_id dataframe into a new dataframe called myDF_species. Print the shape of the dataframe to verify that the column has been extracted. Print the first 5 rows of the dataframe to verify that the columns have been extracted correctly.

We briefly touched on filtering rows based on a condition in Question 4. In this case, we simply filtered by one condition. It is fine to simply filter by one condition repeatedly until you have performed all the filtering you need. However, it is also possible to filter by multiple conditions in a single operation.

Pandas allows us to use logical operators to combine multiple conditions into a boolean expression. The logical operators are & for "and", | for "or", and ~ for "not". We can use these operators allong with conditionals to filter rows based on multiple conditions.

We can store each condition in a variable like so:

Once we have these conditions, we can combine them in the iloc function like so:

Write code to filter the myDF dataframe to only include rows where SepalLengthCm is greater than 5.0, PetalWidthCm is less than 1.5, and Species is not Iris-setosa. Store the filtered dataframe in a new variable called myDF_filtered. How many rows meet these conditions?

One of the most common operations in data analysis is to calculate summary statistics. This includes things like the mean, median, standard deviation, etc.

Pandas has many operations to calculate these statistics. To calculate the mean of a column, we can simply write myDF[column].mean(). Similarly, to calculate the median, we can write myDF[column].median(). For standard deviation, we can write myDF[column].std().

Additionally, we can find all unique values in a column by using the unique function. myDF[column].unique() will return a list containing all unique values in the column.

Write a code to determine the mean, median, and standard deviation of the SepalLengthCm column in the myDF dataframe, for each unique species in the Species column. Please note that there are 3 unique species in the Species column, so you should have 3 sets of statistics.