CS 613: Machine Learning

Spring 2008

image credit: Prof. Mike West, Duke University

---

[Announcements] [Syllabus] [Lectures] [Paper Presentation] [Programming Project] [Resources]

Monday 6:00-8:50PM@University Crossing 153

Ko Nishino
e-mail: kondrexel.edu
office: University Crossing 108
phone: (215) 895-2678
office hours: Tuesday 2:00-3:00 or by e-mail appointment

Prabin Bariya
e-mail: pb339drexel.edu
office: University Crossing 147 (Cyber Learning Center)
phone: (215) 895-2675
office hours: Monday 4:00-6:00 and Wednesday 2:00-4:00 or by e-mail appointment

Louis Kratz
e-mail: lak24drexel.edu
office: Curtis 256
phone: (215) 895-2675
office hours: Thursday 4:00-6:00 or by e-mail appointment

Announcements

[03/28/2008] First class will be 03/31 (Mon)

Syllabus

  Overview

This course introduces modern statistical machine learning in a half-lecture half-seminar format. Lectures will cover the mathematical foundation and representative algorithms of selected topics in machine learning. Throughout the course, strong emphasis will be given to Bayesian modeling and inference. The topics expected to be covered include fundamentals of probabilities and decision theory, regression, classification including support vector machine and relevance vector machine, graphical models including Bayes nets and Markov random field, mixture models, clustering, expectation maximization, hidden Markov models, Kalman filtering, and linear dynamical systems.

Students are expected to give paper presentations on the covered topics and actively participate in discussions on each presentation. Multiple in-class quizzes will also be given to reinforce understanding of each topic. Students are also expected to implement one or more algorithms from the covered topics and apply it to solve a real-world problem using appropriate data.

Students entering the class with pre-existing working knowledge of probability and statistics will be at an advantage but the course is designed so that anyone with strong background or interest in mathematical modeling and analysis can catch up and fully participate. The programming assignment will also be a great chance for students to apply the learnt topics to problems in their own area of interest.

  Prerequisites

There is no official prerequisite for Spring 2008. However, basic (fluency at undergraduate level) understanding of Linear Algebra and Calculus will be necessary. For the programming project, one will need to program in a programming language of their own choice (Matlab, C/C++, Java, etc.). The program must run on tux.cs.drexel.edu.

  Topics

The following is a list of topics and paper presentations that will be covered in this course. The timeline is preliminary and will most likely change.

Week 1	3/31	Introduction, Probabilities
Week 2	4/7	Decision Theory, Regression
Week 3	4/14	Classification
Week 4	4/21	Student Paper Presentations
Week 5	4/28	Graphical Models, Mixture Models
Week 6	5/5	Student Paper Presentations
Week 7	5/12	Sequential Data
Week 8	5/19	Student Paper Presentations
Week 9	5/26	No Class (University Holiday)
Week 10	6/2	Student Presentations (Final Project)

  Textbook

Pattern Recognition and Machine Learning, by Christopher M. Bishop, Springer, 2006. (ISBN-10: 0387310738, ISBN-13: 978-0387310732)
Lectures will folow this book. Drexel Bookstore should have copies.

The following is a list of other textbooks recommended (but not required) for supplemental reading.

Introduction to Machine Learning, by Ethem Alpaydin, The MIT Press, 2004. (ISBN-10: 0262012111, ISBN-13: 978-0262012119)

  Quizzes, Paper Presentation, and Programming Project

Students will take multiple (3 to 5 - TBD) short written exams (quizzes) in class on student paper presentation weeks. These quizzes will mainly cover the material lectured the previous week.

Students will also give one paper presentation. The paper can be chosen from the list of papers compiled by the instructor (see Paper Presentation) or picked from the area of their own interest. For the latter case, the student must consult with the instructor to ensure that the paper is relevant and covers one of the focus topics of the specific week when it will be presented.

Finally, students will also implement one of the algorithms covered in the course and use it to infer meaningful information from data collected by themselves or from public data sets. Students may work on this programming project throughout the entire period of the course and will present the idea, implementation details, and results with appropriate slides on the 10th week. See Programming Project for details.

You must be the sole original author of all assignments and examination solutions in their entirety. As the university's policy explains, penalties up to and including a failing grade for the course with no opportunity to withdraw, will be given for plagiarism, fabrication, or cheating*.
*The standards for originality in a program are similar to those of other written works. Programs by different authors show clear and substantial differences as judged by most criteria, including but not limited to: choice of variable and procedure names, line spacing and indentation, choice of program structure, choice of algorithms, ordering of modules, module design, and ordering and choice of instructions. The original author of an assignment can explain each detail and how they came to create it on their own.

  Grading

The following is tentative and will change as instructor finds necessary.
Paper Presentation: 15%
In-class Quizzes: 45%
Programming Project: 40% (idea, implementation, and presentation)
Assignments turned in up to one day late incur a 50% penalty; assignments turned in more than one day late cannot be accepted and receive a score of 0. Missed quizzes also receive a score of 0. Make-up quizzes will only be allowed in extreme circumstances.

  Communication

The instructor will disseminate important announcements by email through the course mailing list, and also post these announcements on the course web site. Also, the web site contains a time-line with links to all information (lecture slides, assignments, etc.) relevant to the course.

  Policies

Attendance for lectures and exams is expected. In the case of a school closing on an exam day, the exam will be given in the next class period. The Drexel snow emergency information number is (215) 895-6358.

Academic honesty is essential. Cheating, academic misconduct, plagiarism, and fabrication of any submitted material, including both code and prose, are serious breaches of academic integrity and will be dealt with accordingly. Violations will result minimally in a grade of zero for the exam/assignment in question, an additional reduction of one letter grade in the overall course grade, and a report of the violation to the Drexel administration; further penalties may apply to more serious and/or repeat violations. Please refer to Drexel's official Academic Honesty Policy for more information.

Lectures

Paper Presentation

The following is a list of papers for the topics intended to be covered on the corresponding weeks of student paper presentations. These papers are selected from journals and conferences on computer vision and graphics. You may choose a paper not on the list that is in your favorate research area as well. If you choose to do so, please consult with the instructor to make sure that the paper of your choice actually covers the intended topic of the week!

Week 4: Regression and Classification

(Maximum Likelihood, Bayesian Regression, Support Vector Machine, Relevance Vector Machine)

• A Bayesian Approach to Digital Matting, Y-Y. Chuang, B. Curless, D.H. Salesin, and R. Szeliski, CVPR 2001.
• Object Class Recognition by Unsupervised Scale-Invariant Learning, R. Fergus, P. Perona, and A. Zisserman, CVPR 2003.
• Image Retrieval and Classification Using Local Distance Functions, A. Frome, Y. Singer, and J. Malik, NIPS 2006.
• Gender Classification with Support Vector Machines, B. Moghaddam and M-H. Yang, FG 2000. (Sukhbir Singh)
• Bayesian Face Recognition, B. Moghaddam, T. Jebara, and A. Pentland, PR 2000. (Geoff Oxholm)
• Deriving Intrinsic Images from Image Sequences, Y. Weiss, ICCV 2001.
• Learning Image Statistics for Bayesian Tracking, H. Sidenbladh and M.J. Black, ICCV 2001. (Louis Kratz)
• Support Vector Tracking, S. Avidan, PAMI 2004.
• Bayesian Image Segmentation Using Wavelet-Based Priors, CVPR 2006. (Peter Bogunovich)

Week 6: Graphical Models and Mixture Models

(Bayes Nets, Markov Random Field, Clustering)

• Robustly Estimating Changes in Image Appearance, M.J. Black, D.J. Fleet, and Y. Yacoob, CVIU 2000.
• Dynamic Textures, G. Doretto, A. Chiuso, Y.N. Wu, and S. Soatto, IJCV 2003.
• Learning Low-Level Vision, W.T. Freeman, E.C. Pasztor, and O.T. Carmichael, IJCV 2000.
• Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images, S. Geman and D. Geman, PAMI 1984.
• Clustering Appearance and Shape by Learning Jigsaws, A. Kannan, J. Winn, and C. Rother, NIPS 2006.
• Learning Depth from Single Monocular Images, A. Saxena, S.H. Chung, and A.Y. Ng, NIPS 2005. (Linge Bai)
• Video Google: A Text Retrieval Approach to Object Matching in Videos, J. Sivic and A. Zisserman, ICCV 2003. (Lingchuan Meng)
• MAP Source Separation using Belief Propagation Networks, SSC 2006. (Dmitriy Bespalov)
• Ad hoc Networking, Markov Random Fields, and Decision Making, SP 2006. (Robert Lass)

Week 8: Sequential Data

(Hidden Markov Model, Kalman Filtering, Linear Dynamical Systems)

• Unsupervised Scene Analysis: A Hidden Markov Model Appraoch, M. Bicego, M. Cristani, and V. Murino, CVIU 2006. (Kamelia Aryafar)
• Dining Activity Analysis Using a Hidden Markov Model, J. Gao, A.G. Hauptmann, A. Bharucha, and H.D. Wactlar, ICPR 2004.
• Transformed Hidden Markov Models: Estimating Mixture Models of Images and Inferring Spatial Transformations in Video Sequences, N. Jojic, N. Petrovic, B.J. Frey, and T.S. Huang, CVPR 2000.
• Motion Texture: A Two-Level Statistical Model for Character Motion Synthesis, Y. Li, T. Wang, and H-Y. Shum, SIGGRAPH 2002. (Daniel O'Callaghan-Horn)
• Learning and Detecting Activities from Movement Trajectories Using the Hierarchical Hidden Markov Model, N.T. Nguyen, D.Q. Phung, S. Venkatesh, and H. Bui, CVPR 2005. (Paul Snyder)
• Parametric Hidden Markov Models for Gesture Recognition, A.D. Wilson and A.F. Bobick, PAMI 1999.
• 3D Object Recognition Using Bayesian Geometric Hashing and Pose Clustering, A. Seghal and U.B. Desai, Pattern Recognition 2002. (Prabin Bariya)

Programming Project

Students will be required to implement a discussed machine learning technique of their choosing. Students are free to choose their approach and models based on the machine learning problem they wish to investigate. Suggested data sets are provided below, however students are encouraged to apply machine learning techniques to their own area of research or interest. The specific machine learning technique must be implemented by the student, in the programming environment of their choosing. Other libraries may be used (for data I/O, result analysis, etc), however the core algorithm must be implemented by the student alone.

The final project submission must include a formal analysis of the proposed method, data, and results in the form of a NIPS paper (templates available here). The write up must include an analysis of your work's approach, and its performance under varying conditions. Students are encouraged to vary elements of their approach and measure the results. Elements may include different features, distribution functions, or data sets.

Project Ideas and Data Sets

The following is some links to data sets mainly from a machine learning course at CMU by Professor Carlos Guestrin.

• Brain Image Data: CMU provides excellent ideas and data for brain imaging.
• The NetFlix Challenge: Netflix provides a downloadable database of user ratings for movie titles. They have an ongoing contest to predict user ratings of films.
• Sensor Network Data: This very real world data set provides interesting challenges.
• Image Data: CMU presents data for vision problems such as object recognition, face recognition, image segmentation, and face recognition.
• Other Data Sets: UCI has a large collection of data sets for machine learning.

Many of these pages have suggested problems to run on the data sets. Students are encouraged to think of their own problems to run on the data sets where possible (For example, detecting facial expressions in the face data set, or commercial detection on video data sets).

Resources