DATA 3464: Fundamentals of Data Processing on DATA 3464

Lab 10: Image files and processing

Mon, 30 Mar 2026 00:00:00 +0000

Lab 10: Image files and processing#

Objective#

Learn about:

Images as 2D signals
Impact of resampling approaches
Bulk image processing

Setup#

The lab directory has some starter code and files in the lab10 subdirectory, so merge in the pull request and pull changes to your local computer.

Task 1: Resizing#

One of the most common image processing operations is shrinking (or, less commonly, increasing) the number of pixels. Even with libraries that provide convenient “resize” functions, you need to choose the resampling algorithm and the target dimensions.

Lab 9: 1D Signals and Audio

Mon, 23 Mar 2026 00:00:00 +0000

Lab 9: 1D Signals and Audio#

Objective#

Learn about:

Audio as 1D signals
Time and frequency representations
Characteristics of audio files
Basic audio processing

Setup#

The lab directory has some starter code and files in the lab09 subdirectory, so merge in the pull request and pull changes to your local computer.

I’ve downloaded two files from freesound.org. Do not listen to them - try to figure out which is which by looking at the signals! The files are:

Lab 8: Bash and Regular Expressions

Mon, 16 Mar 2026 00:00:00 +0000

Lab 8: Bash and Regular Expressions#

Objective#

Learn about:

Navigating a computer on the command line
Basic bash syntax
Extracting text features with regular expressions

Setup#

There is no starter code this week; instead, I’d like you to create a lab08 subdirectory and put your files in it.

While I can’t tell how you actually make a directory, this is a good excuse to practice with bash! Try doing the following:

Assignment 3: Dataset curation

Wed, 11 Mar 2026 00:00:00 +0000

Assignment 3: Dataset curation#

Due April 2, 2026 (end of day). This is the day before Easter break; presentations are tentatively scheduled for the following Thursday (April 9). As usual, reasonable requests for extensions will be granted.

You may work in groups up to 3, and I strongly advise working in groups this time! Some of the work is just plain tedious. Click here to create your groups on GitHub Classroom and clone your mostly empty repository.

Lab 7: Text Wrangling

Mon, 09 Mar 2026 00:00:00 +0000

Lab 7: Text Wrangling#

Objective#

Learn a bit about:

Text files that don’t magically import
Converting from text to datetime and numeric formats

Setup#

Update your labs repo through the usual method. There is one starter notebook with guiding TODO comments and some data files.

Your task#

This activity was inspired by a friend who complained that Canada Pension Plan (CPP) contribution maximums have been outpacing inflation (yes, this is very much a first world problem). The CPP website provides information in tabular form, but it’s nice to interpret it visually.

Lab 6: Lab Exam Trial

Mon, 23 Feb 2026 00:00:00 +0000

Lab 6: Lab Exam Trial#

Next week (Monday, March 2th) we’ll be doing the lab component of the midterm, worth 5-10% of your final grade (whichever gives you a higher overall score with the written midterm at 15-20%). To test out the setup in a low stakes environment, this lab will be conducted in class with the following websites whitelisted:

The usual lab computer environments will be available - VS Code, Jupyter Notebooks, Spyder, etc. There is currently an issue with Python’s intellisense extension on VS Code (Pylance), but hopefully this will be resolved before the real thing. In the meantime, enjoy the challenge of writing code without autocomplete.

Assignment 2: Preprocessing Pipelines

Mon, 09 Feb 2026 00:00:00 +0000

Assignment 2: Preprocessing Pipelines#

Due February 24, 2026 (end of day). Reasonable requests for extensions will be granted. This is, awkwardly, only 2 days before the midterm. I will put a component of this assignment on the lab component of the midterm (March 9 2), so timely submissions will ensure I can give you feedback before then.

You may work in groups up to 3. Click here to create your groups on GitHub Classroom and clone the starter repository, which should have a csv containing the data and the usual .gitignore file.

Lab 5: Dimensionality Reduction

Mon, 09 Feb 2026 00:00:00 +0000

Lab 5: Dimensionality Reduction#

Objective#

Learn a bit about:

Dimensionality reduction
The differences between PCA and LDA

Setup#

Update your labs repo through the usual method. There is one starter notebook that just fetches the dataset and has some guiding TODO comments.

The dataset#

For the purposes of this exercise, we’ll use another well-behaved dataset: the Winsconsin Breast Cancer Diagnostics dataset. This has a set of 30 measurements taken from images of histology slides of breast mass biopsies (the features) that can be used to predict whether the mass is benign (0) or malignant (1). You can read the original paper here if you’re into that sort of thing.

Lab 4: Numeric Data Transformations

Fri, 30 Jan 2026 18:03:30 -0700

Lab 4: Numeric Data Transformations#

Objective#

Learn a bit about:

Transforming numeric data
Building preprocessing pipelines

Since you’ve already done a lot of painful wrangling in your first assignment to combine data into a useful tabular form, I’ve done this bit for you in this lab. We’re also going to use the same (sort of) housing assessment data from lab 2, so the dataset should be familiar.

If you are unable to fetch data from the City: I’ve put a copy of a (200MB+) CSV version at “I:\Labs\CompSci\Resources\DATA 3464\housing_data_pre_split.csv”, accessible either through the lab computers or WebFiles at gp.mtroyal.ca

Assignment 1: Exploratory Data Analysis

Thu, 15 Jan 2026 00:00:00 +0000

Assignment 1: Exploratory Data Analysis#

Due January 30, 2026 at 5 pm. Reasonable requests for extensions will be granted.

You may work in groups up to 3. Click here to create your groups on GitHub Classroom and clone the starter repository, which is pretty much empty aside from a .gitignore file. This should allow everyone in the group to make changes, but merge conflicts may occur, particularly if two people are editing the same file.

12: Data Labelling and Augmentation

Thu, 02 Apr 2026 00:00:00 +0000

Topic overview#

Tools for fancy labelling
Annotation conventions
Augmenting data

Resources used:

Computer vision tasks#

How are annotations stored?#

Usually in plain text!

Classification: subdirectories, csv files
Bounding boxes: text files, e.g. COCO, VOC
Segmentations: Text and/or PNG, e.g. COCO

Not a lot of formal process here - someone builds something for their purposes, others find it useful, variations abound.

11: Image Processing

Tue, 24 Mar 2026 00:00:00 +0000

Topic overview#

Image representation
File formats and compression
Preprocessing for image recognition tasks

Resources used:

Images as 2D signals#

The light that enters a camera can be modelled as continuous signal: $$f(x, y),\space -\infty < x, y < \infty$$
Digital images are sampled: $$f[n, m], \space n = n \Delta_x, m = n \Delta_y$$ where typically $\Delta_x = \Delta_y$
The area $\Delta_x \times \Delta_y$ is called a picture element, or pixel

10: Signals and Audio

Tue, 17 Mar 2026 00:00:00 +0000

Topic overview#

Introduction to signals
Audio as a 1D signal
File formats
A brief intro to signal processing

Resources used:

Various textbooks from my undergrad
DSPguide.com seems like a pretty good resource

What is a signal?#

“A [continuous/discrete] signal is a function of independent variables that range over [a continuum/discrete] values” - Jerry L. Prince, Medical Imaging Signals and Systems

9: Bash and data cards

Thu, 12 Mar 2026 00:00:00 +0000

Topic overview#

Intro to bash and unix tools
Data cards

Resources used:

Why these topics together? They both seem relevant to assignment 3

Why bash?#

Python is great, but it’s not the only tool. Command line is better to:
- Move and rename files
- Take a peek at the first few lines of a giant csv
- Find and replace text in a bunch of files
- Fetch data from the web
Bash and unix tools are more useful than their Windows counterparts
Git bash lets us use bash on Windows

Okay, but what is it?#

The Bourne Again SHell is a command line interface created in 1989
A shell is a program to execute commands from the user
The “Bourne shell” (sh) from the 1970s was standard on UNIX, bash adds to it
Default shell for linux, (almost) macOS
When you run these commands in Jupyter notebook, you’re using bash!
```
%pip install -q some_package
```

Basic bash#

command_name --long_flag -l -o arg1 arg2

Examples of common commands:
- ls to list files in a directory
- cd to change directories
- mv to move or rename files
- head to view the first few lines of a file
- grep to search for text in files
- sed to find and replace text in files
- curl to fetch data from the web

Variables#

Variables are defined with = and accessed with $

8: Wrangling text

Thu, 05 Mar 2026 00:00:00 +0000

Topic overview#

Text file formats and character encodings
Extracting features from (un)structured text
Regular expressions

Resources used:

What is a text file?#

Any file is just a sequence of bytes
The file extension is somewhat meaningless
Text files contain only human-readable characters
Binary files are everything else, including:
- Images
- PDFs
- Word documents*
- Executables

Character encodings#

7: Interaction effects

Thu, 12 Feb 2026 00:00:00 +0000

Topic overview#

Definitions and description of interaction effects
Detecting interaction effects
A brief discussion on feature selection

Resources used:

Definition#

Two or more predictors are said to interact if their combined effect is different (less or greater) than what we would expect if we were to add the impact of each of their effects when considered alone. – Feature Engineering, Ch 7

6: Missing and weird data

Thu, 05 Feb 2026 00:00:00 +0000

Topic overview#

What to do with missing data
Detecting and handling outliers

Resources used:

Feature Engineering Chapter 8
Hands on Machine Learning with Scikit-Learn and Tensorflow/PyTorch, Chapter 2. Available at MRU Library
Scikit-learn user guide: Chapters 2 and 7

The problem#

As you’ve seen, real-world data is messy
Missing values are common, other values don’t make sense
We need to decide how to deal with these problems

What examples have we seen so far? Why might data be missing or weird*?

5: Numeric transformations

Tue, 27 Jan 2026 00:00:00 +0000

Topic overview#

Why transformations are necessary
Common transformations
Dimensionality reduction

Resources used:

Feature Engineering Chapter 6
Hands on Machine Learning with Scikit-Learn and Tensorflow/PyTorch, Chapter 4. Available at MRU Library
Scikit-learn user guide: Chapter 7
Introduction to Machine Learning with Python. Available at MRU Library

Common 1:1 transformations#

“Most models work best when each feature (and in regression also the target) is loosely Gaussian distributed” – Introduction to Machine Learning with Python

Lab 3: Basic supervised models

Mon, 26 Jan 2026 00:00:00 +0000

Lab 3: Basic supervised models#

Objective#

Learn a bit about:

Different processing needs for different model types
The need for validation data
The overall iterative workflow of making predictions from data

As we haven’t yet talked about feature engineering and numeric data transformations, the focus will be on the categorical data.

Setup#

Eventually I’m going to stop posting this, but just like last week, update your labs repo by opening it on GitHub and following the instructions in the readme file. This should result in the lab03 folder with two csvs being added. To download to your local machine, you may need to commit all local changes first and then run git pull.

4: Categorical data

Tue, 20 Jan 2026 00:00:00 +0000

Topic overview#

Exploring categorical data
Categorical data encoding strategies

Resources used:

What is categorical data?#

Samples can take on one of several discrete values or groups
- Nominal: no particular order to the groups
- Ordinal: groups relate to each other in a specific order
Categories can be represented as strings or numeric types
- Domain knowledge is necessary!

Let’s take a few minutes to brainstorm some examples

Lab 2: Fetching and exploring data

Mon, 19 Jan 2026 00:00:00 +0000

Lab 2: Fetching and exploring data#

Objective#

I’ve prepared a small exercise to work with City of Calgary data. This is relevant to your first assignment, and is an example of the widely-used OAuth protocol. Admittedly, I only understand enough of this protocol to be somewhat dangerous.

Setup#

First, update your labs repo by opening it on GitHub and following the instructions in the readme file. This should result in the lab02 folder with starter code being added. To download to your local machine, you may need to commit all local changes first and then run git pull.

3. Exploratory Data Analysis

Thu, 15 Jan 2026 00:00:00 +0000

Topic overview#

Exploratory data analysis (EDA)
Splitting your data

Resources used:

Feat.Engineering Chapter 3
R for Data Science (2e), Chapter 10
Hands on Machine Learning with Scikit-Learn and Tensorflow/PyTorch, Chapter 2. Available at MRU Library

Exploratory data analysis#

The goal of EDA is to Understand your data

Ask questions (e.g. are my data normally distributed?)
Look for answers (e.g. by making histograms)
Find more questions and return to step 1 (hmm, those are some weird numbers, what do these values represent?)

EDA is not a formal process with a strict set of rules. More than anything, EDA is a state of mind. During the initial phases of EDA you should feel free to investigate every idea that occurs to you. Some of these ideas will pan out, and some will be dead ends. – Hadley Wickham

2. Basic machine learning models

Tue, 13 Jan 2026 00:00:00 +0000

Topic overview#

Some common machine learning tasks and models
Evaluating model performance
Limitations and assumptions

Resources used:

Feature Engineering Chapter 1
Introduction to Machine Learning with Python. Available at MRU Library
Scikit-learn User Guide
Hands on Machine Learning with Scikit-Learn and Tensorflow/PyTorch. Available at MRU Library

Machine learning#

To appropriately process the data, we need to know why we are doing it and what assumptions we’re making
Modern machine learning toolkits (such as scikit-learn) are so easy to use, they’re easy to use inappropriately
Goal: just enough understanding to use basic models responsibly

Why are we processing data?#

Lab 1: Regression with Clean Data

Mon, 12 Jan 2026 00:00:00 +0000

Lab 1: Regression with Clean Data#

Due date: January 19, 2026 (next lab session)

Objective#

The main goal of this exercise is for me to learn about your current habits and knowledge. It also serves as an example for why we need “clean” and easy to manage data.

Deliverables#

Click here to join the GitHub classroom and clone the “starter code”. There’s not much in here other than the data sets, so you’ll need to add:

1. Introduction

Tue, 06 Jan 2026 00:00:00 +0000

Meet your instructor#

Name: Charlotte Curtis

Pronouns: She/her

Office: B102-4

Email: ccurtis@mtroyal.ca

Office hours: Book here

My Background#

Another new class!#

This course introduces techniques for ethically and responsibly wrangling and manipulating datasets to make them appropriate for addressing the question at hand. Topics may include cleaning and transforming data, integrity and quality measures, common file formats, feature selection and engineering, and generating features from unstructured sources such as text and images.