Course information

Statistical learning is about the construction and study of systems that can automatically learn from data. With the emergence of massive datasets commonly encountered today, the need for powerful machine learning is of acute importance. Examples of successful applications include effective web search, anti-spam software, computer vision, robotics, practical speech recognition, and a deeper understanding of the human genome. This course gives an introduction to this exciting field, with a strong focus on kernels as a versatile tool to represent data, in combination with (un)supervised learning techniques that are agnostic to the type of data that is learned from. The learning techniques that will be covered include regression, classification, clustering and dimension reduction. We will cover both the theoretical underpinnings of kernels, as well as a series of kernels that are important in practical applications.


  • For UJF: homeworks 1,2,3 (1/2) + project (1/2)
  • For ENSIMAG: homework 1 (1/4) + project (3/4)

Course outline


  • Motivating example applications
  • Empirical risk minimization
  • Bias-variance trade-off, and risk bounds

Supervised learning with linear models and kernels

  • Risk convexification and regularization
  • Ridge regression
  • Logistic regression
  • Support vector machines
  • Kernels for non-linear models

Unsupervised learning

  • Principal component analysis
  • Data clustering
  • Other methods: canonical correlation analysis, sparse coding, etc.

Kernels for probabilistic models

  • Fisher kernels
  • Probability product kernels

Reading material

Machine Learning and Statistics

  • Vapnik, The nature of statistical learning theory. Springer
  • Hastie, Tibshirani, Friedman, The elements of statistical learning. (free online)
  • Devroye, Gyorfi, Lugosi, A probabilistic theory of pattern recognition. Springer
  • J Shawe-Taylor, N Cristianini. Kernel methods for pattern analysis. 2004.
  • Bishop, Pattern recognition & machine learning. 2006.
  • Slides by Jean-Philippe Vert on kernel methods.


  • S. Boyd and L. Vandenberghe. Convex Optimization. 2004. (free online)
  • D. Bertsekas. Nonlinear Programming. 2003.


Date Room Lecturer Topic Homework
07/10 H104 JV Introduction + Bias-variance tradeoff.
14/10 H201 JV Penalized empirical risk minimization, linear classifiers, introduction kernels.
Homework 1
21/10 H201 JM Reproducing kernel Hilbert spaces (RKHS)
04/11 H201 JV The kernel trick, supervised kernel methods, and Fisher kernels.
Homework 2
25/11 H201 JM
2/12 H201 JM


There will be three homeworks given during the course, at lecture 2, 4, and 6. Each of them should be returned within three weeks. Either use LateX, or make sure you write very clearly. Homework has to be done individually. ENSIMAG students only have to handin the first homework, since they get less credits for the course.


The project consists of implementing an article, doing some experiments, and writing a small report (less than 10 pages). It is also possible to study a theoretical paper instead of implementing a method. All reports should be written in LateX, and a pdf should be sent to the lecturers before January 5th. Projects can be done alone, or in groups of two people. You can either come with your own idea and discuss it with us, or we can give you some suggestions. To give you an idea, these are projects of a related course.
Project Student(s) Coach
Supervised classification of text documents.
Vera Shalaeva and Manon Lukas
Predicting Molecular Activity with Graph Kernels.
Phivos Valougeorgis
Speaker Recognition.
Li Liu
Supervised classification of Flickr images.
Leonardo Gutierrez Gomez
Fast string kernels using inexact matching for protein sequences
Semigroup kernels on measures
Julien Alapetite
Kernel change-point analysis
Fast global alignment kernels
Multiple kernel learning, conic duality, and the SMO algorithm
Predictive low-rank decomposition for kernel methods
Image classification with segmentation graph kernels
Image Classification with the Fisher Vector: Theory and Practice
Jerome Lesaint

Jobs / Internships Opportunities

We have different intern/PhD opportunities in machine learning, image processing, bioinformatics and computer vision. It is best to discuss that matter early with us since the number of places is limited.