Section outline

  • Intelligent Systems for Pattern Recognition - 6CFU - DEPRECATED SINCE 2022

    Collapse all Expand all

    Code: 651AA, Credits (ECTS): 6, Semester: 2, Official Language: English

    Instructor: Davide Bacciu

    Contact: email - phone 050 2212749

    Office: Room 367, Dipartimento di Informatica, Largo B. Pontecorvo 3, Pisa

    Office Hours: Thursday, 16-18 (email to arrange meeting)


  • News

    Highlighted

  • Course Information

    Weekly Schedule

    The course is held on the second term. The schedule for A.A. 2020/21 is provided in table below.

    The first lecture of the course will be on Thursday 18/02/2021. The course will be offered online only due to the COVID19 pandemic. Lectures will be streamed following the schedule below by leveraging the dedicate MS Team accessible with this link.

    Recordings of the lectures will be made available to the students following the course.

    Day Time
    Thursday
    		 14.15-16.00
    Friday
    		 11.00-12.45


    Objectives

    Course Prerequisites

    Course prerequisites include knowledge of machine learning fundamentals (e.g. covered through ML course). Knowledge of elements of probability and statistics, calculus and optimization algorithms are also expected. Previous programming experience with Python is a plus for the practical lectures.

    Course Overview

    The course introduces students to the analysis and design of advanced machine learning models for modern pattern recognition problems and discusses how to realize advanced applications exploiting computational intelligence techniques.

    The course is articulated in four parts. The first part introduces basic concepts and algorithms concerning pattern recognition, in particular as pertains sequence and image analysis. The next two parts introduce advanced models from two major learning paradigms, that are (deep) neural networks and generative models, and their use in pattern recognition applications. The last part will go into the details of the realization of selected recent applications of AI models. Presentation of the theoretical models and associated algorithms will be complemented by introductory classes on the most popular software libraries used to implement them.

    The course hosts guest seminars by national and international researchers working on the field as well as by companies that are engaged in the development of advanced applications using machine learning models.

    Topics covered - Bayesian learning, graphical models, deep learning models and paradigms, deep learning for machine vision and signal processing, advanced neural network models (recurrent, recursive, etc.), (deep) reinforcement learning, signal processing and time-series analysis, image processing, filters and visual feature detectors, pattern recognition applications (machine vision, bio-informatics, robotics, medical imaging, etc), introduction to machine learning and deep learning libraries.

    Textbook and Teaching Materials

    The course does not have an official textbook covering all its contents. However, a list of reference books covering parts of the course is listed at the bottom of this section (note that all of them have an electronic version freely available online).

    [BRML] David Barber, Bayesian Reasoning and Machine Learning, Cambridge University Press

    [DL] Ian Goodfellow and Yoshua Bengio and Aaron Courville , Deep Learning, MIT Press

  • Lectures and Calendar

    The official language of the course is English: all materials, references and books are in English.

    Lecture slides will be made available here, together with suggested readings.

    Date Room Topic References Additional Material
    1 18/02/2021
    (14-16)
    		 ONLINE Introduction to the course: motivations and aim; course housekeeping (exams, timetable, materials); introduction to modern pattern recognition applications
    (slides)




    219/02/2021
    (11-13)    		ONLINESignal processing: timeseries; time domain analysis (statistics, correlation); spectral analysis; fourier analysis.
    (slides)


    3
    		25/02/2021
    (14.00-15.30)
    		ONLINE
    		Image processing: feature descriptors (color histograms, SIFT), spectral analysis, feature detectors (edge, blobs and segments).
    (slides)

    		Additional Readings
    [1,2] Two high-level surveys on visual feature extraction and representation
    4
    		26/02/2021
    (11-13)
    		ONLINE
    		Generative and Graphical Models - Part 1: probability refresher; graphical model representation; directed and undirected models
    (slides)
    		[BRML] Ch. 1 and 2 (Refresher)
    [BRML] Sect. 3.1, 3.2 and 3.3.1
    (conditional independence)

    5
    		 04/03/2021
    (14-16)
    		 ONLINE
    		 Generative and Graphical Models - Part 2: Bayesian networks; Markov networks; conditional independence and d-separation.
    		 [BRML] Sect. 3.3 (Directed Models)
    [BRML] Sect. 4.1, 4.2.0-4.2.2 (Undirected Models)
    [BRML] Sect. 4.5 (Expressiveness)
    		 Software

    		05/03/2021
    (11-13)
    		NO LECTURE DUE TO GRADUATION COMMITTEE (WILL BE RECOVERED)

    6
    		 11/03/2021
    (14-16)
    		 ONLINE
    		 Hidden Markov Models  - Part 1:  learning in directed graphical models; forward-backward algorithm;  generative models for sequential data
    (slides)
    		 [BRML] Sect. 23.1.0 (Markov Models)
    [BRML] Sect. 23.2.0-23.2.4 (HMM and forward backward) 
    		 Additional Readings
    [3]  A classical tutorial introduction to HMMs
    7
    		 12/03/2021
    (11-13)
    		 ONLINE
    		 Hidden Markov Models - Part 2: EM algorithm, learning as inference, Viterbi Algorithm
    		[BRML] Sect. 23.2.6 (Viterbi)
    [BRML] Sect. 23.3.1-23.3.4 (EM and learning)
    8
    		 18/03/2021
    (14-16)
    		 ONLINE
    		 Markov Random Fields: learning in undirected graphical  models; conditional random fields; pattern recognition applications
    (slides)
    		 [BRML] Sect. 4.2.2, 4.2.5 (MRF)
    [BRML] Sect. 4.4 (Factor Graphs)
    [BRML] Sect. 5.1.1 (Variable Elimination and Inference on Chain) 
    [BRML] Sect. 9.6.0, 9.6.1, 9.6.4, 9.6.5 (Learning in MRF/CRF)
    		 Additional Readings
    [4,5] Two comprehensive tutorials on CRF ([4] more introductory and [5] more focused on vision)
    [6] A nice application of CRF to image segmentation

    Sofware
    Check out pgmpy: it has Python notebooks to introduce to working with MRF/CRF
    9
    		 19/03/2021
    (11-13)
    		 ONLINE
    		 Boltzmann Machines: bridging neural networks and generative models; stochastic neuron; restricted Boltzmann machine; contrastive divergence
    (slides)
    		  [DL] Sections 20.1 and 20.2
    		 Additional Readings
    [7] A clean and clear introduction to RBM from its author

    Sofware
    Matlab code for Deep Belief Networks (i.e. stacked RBM) and Deep Boltzmann Machines.
    10
    		 25/03/2021
    (14-16)
    		 ONLINE
    		 Bayesian Learning: non-parametric models; variational learning
    (slides)
    		[BRML] Sect. 11.2.1 (Variational EM), 20.4-20.6.1  (LDA)
    		 Additional Readings
    [8] LDA foundation paper
    [9] A gentle introduction to latent topic models
    [10] Foundations of bag of words image representation

    Sofware
    11
    		 26/03/2021
    (11-13)
    		 ONLINE
    		 Bayesian Learning: sampling methods - Guest lecture by Daniele Castellana
    (    slides)
    		 [BRML] Sect. 27.1-27.3, 27.4.1, 27.6.2
    M1
    		 30/03/2021
    (13.30-16.00)
    		 ONLINE
    		 Midterm 1 discussions

    12
    		 08/04/2021
    (14-16)
    		 ONLINE
    		 Convolutional Neural Networks (part I): introduction to CNN; basic CNN elements
    		 [DL] Chapter 9
    		 Additional Readings
    [11-15] Original papers for LeNet, AlexNet, VGGNet, GoogLeNet and ResNet.
    13
    		 09/04/2021
    (11-13)
    		 ONLINE
    		 Convolutional Neural Networks (part II): CNN architectures for image recognition; convolution visualization; advanced topics (deconvolution, dense nets); applications and code
    (slides)
    		 [DL] Chapter 9
    		 Additional Readings
    [16] Complete summary of convolution arithmetics
    [17] Seminal paper on batch normalization
    [18] CNN interpretation using deconvolutions
    14
    		 15/04/2021
    (14-16)
    		 ONLINE
    		 Deep Autoencoders: introduction to the deep learning module, sparse, denoising and contractive AE; deep RBM
    (slides)
    		 [DL] Chapter 14, Sect 20.3, 20.4.0 (from 20.4.1 onwards not needed)
    		 Additional Readings
    [19] DBN: the paper that started deep learning
    [20] Deep Boltzmann machines paper
    [21] Review paper on deep generative models
    [22] Long review paper on autoencoders from the perspective of representation learning
    [23] Paper discussing regularized autoencoder as approximations of likelihood gradient
    15
    		 16/04/2021
    (11-13)
    		 ONLINE
    		 Gated Recurrent Networks: deep learning for sequence processing; gradient issues; long-short term memory; gated recurrent units; generative use of RNN
    (slides)
    		 [DL] Sections 10.1-10.3, 10.5-10.7, 10.10, 10.11
    		 Additional Readings
    [24] Paper describing gradient vanish/explosion
    [25] Original LSTM paper
    [26] An historical view on gated RNN
    [27] Gated recurren units paper
    [28] Seminal paper on dropout regularization

    Sofware
    16
    		 22/04/2021
    (14-16)
    		 ONLINE
    		 Gated Recurrent Networks II

    17
    		 23/04/2021
    (11-13)
    		 ONLINE
    		 Deep randomized neural networks - Guest lecture by Claudio Gallicchio
    (slides)

    18
    		 29/04/2021
    (14-16)
    		 ONLINE
    		 Learning in Structured Domain I: Recursive Neural Networks - Guest lecture by Alessio Micheli
    (slides)

    19
    		 30/04/2021
    (11-13)
    		 ONLINE
    		 Learning in Structured Domain II: Neural Networks for Graphs - Guest lecture by Alessio Micheli
    (slides)

    M2
    		 03/05/2021
    (13.30-17)
    		 ONLINE
    		 Midterm 2 discussions

    20
    		 06/05/2021
    (14-16)
    		 ONLINE
    		 An introduction to Tensorflow and Keras: python; numpy, tensorflow, keras;
    (Coding practice by Federico Errica)
    (slides)

    21
    		 07/05/2021
    (11-13)
    		 ONLINE
    		 PyTorch – neural networks in Python: python; pytorch; RNN
    (Coding practice by Antonio Carta)
    (slides)

    22
    		 13/05/2021
    (14-16)
    		 ONLINE
    		 Advanced Recurrent Architectures: sequence-to-sequence;  attention models; multiscale network; memory networks; neural reasoning.
    (slides)


    		 [DL] Sections 10.12, 12.4.5
    		 Additional Readings
    [29,30] Models of sequence-to-sequence and image-to-sequence transduction with attention
    [31,32] Models optimizing dynamic memory usage (clockwork RNN, zoneout)
    [33] Differentiable memory networks
    [34,35] Neural Turing Machines and follow-up paper on pondering networks
    [36] Transformer networks: a paper on the power of attention without recurrence
    23
    		 14/05/2021
    (11-13)
    		 ONLINE
    		 Generative and Unsupervised Deep Learning: explicit distribution models; variational autoencoders; adversarial learning.
    (slides)
    		 [DL] Sections 20.9, 20.10.1-20.10.4    
    		Additional Readings
    [37] PixelCNN - Explict likelihood model
    [38] Tutorial on VAE
    [39] Tutorial on GAN (here another online resource with GAN tips)
    [40] Wasserstein GAN
    [41] Tutorial on sampling neural networks

    Sofware
    24
    		 20/05/2021
    (14-16)
    		 ONLINE
    		 Continual learning. Guest lecture by Vincenzo Lomonaco
    25
    		 21/05/2021
    (11-13)
    		 ONLINE
    		 Final lecture: course wrap-up; research themes; final projects; exam modalities
    (slides)

     M3  07/06/2021
    (10-13.30)
    		 ONLINE
    		 Midterm 3 discussions
    		    

  • Course Grading and Exams

    Typical course examination (for students attending the lectures) is performed in 2 stages: midterm assignments and an oral presentation. Midterms waive the final project.

    Midterm Assignment

    Midterm assignments consist in a very short presentation (5 minutes per person) to be given in front of the class, presenting the key/most-interesting aspects of one of the following tasks:

    • A quick and dirty (but working) implementation of a simple pattern recognition algorithm
    • A report concerning the experience of installing and running a demo application realized using available deep learning and machine learning libraries
    • A summary of a recent research paper on topics/models related to the course content.

    Students might be given some amount of freedom in the choice of assignments, pending a reasonable choice of the topic. The assignments will roughly be scheduled every 1 month, with the last one being performed the day of the oral examination. 

    Oral Exam

    The oral examination will test knowledge of the course contents (models, algorithms and applications).

    Grading

    The final exam vote \(G\) is determined as

     \( G = G_O + \sum_{i=1}^{4} G_{M}^{i} \)

    where

    • \( G_O \in [1,21] \) is the oral grade
    • \( G_{M}^i \in [0,3] \) is the grade for the i-th midterm

    Alternative Exam - Non attending students

    Working students, those not attending lectures, those who have failed or are unsatisfied with midterms can complete the course by delivering a final project and an oral exam.  Contact the instructor by mail to arrange project topics and examination dates.

    The final project concerns preparing a report on a topic relevant to the course content or the realization of a software implementing a non-trivial learning model and/or a PR application relevant for the course. The content of the final project will be discussed in front of the instructor and anybody interested during the oral examination. Students are expected to prepare slides for a 20 minutes presentation which should summarize the ideas, models and results in the report. The exposition should demonstrate a solid understanding of the main ideas in the report.

    Grade for this exam modality is determined as

     \( G = 0.5 \cdot (G_P + G_O) \)

    where \( G_P \in [1,32] \) is the project grade and \( G_O \in [1,30] \) is the oral grade

  • Midterms and Projects

  • Past Editions

  • References

    Bibliographic References
    1. Scott Krigg, Interest Point Detector and Feature Descriptor Survey, Computer Vision Metrics, pp 217-282, Open Access Chapter
    2. Tinne Tuytelaars and Krystian Mikolajczyk, Local Invariant Feature Detectors: A Survey, Foundations and Trends in Computer Graphics and Vision, Vol. 3, No. 3 (2007) 177–2, Online Version
    3. Lawrence R. Rabiner:a tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 1989, pages 257-286, Online Version
    4. Charles Sutton and Andrew McCallum,  An Introduction to Conditional Random Fields, Arxiv
    5. Sebastian Nowozin and Christoph H. Lampert, Structured Learning and Prediction, Foundations and Trends in Computer Graphics and Vision, Online Version
    6. Philipp Krahenbuhl, Vladlen Koltun, Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials, Proc.of NIPS 2011, Arxiv
    7. Geoffrey Hinton, A Practical Guide to Training Restricted Boltzmann Machines, Technical Report 2010-003, University of Toronto, 2010
    8. D. Blei, A. Y. Ng, M. I. Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 2003
    9. D. Blei. Probabilistic topic models. Communications of the ACM, 55(4):77–84, 2012, Free Online Version
    10. G. Csurka, C. R. Dance, L. Fan, J. Willamowski, and C. Bray. Visual Categorization with Bags of Keypoints. Workshop on Statistical Learning in Computer Vision. ECCV 2004, Free Online Version
    11. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel. Handwritten digit recognition with a back-propagation network, Advances in Neural Information Processing Systems, NIPS, 1989
    12. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems, NIPS, 2012
    13. S. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition, ICLR 2015, Free Online Version
    14. C. Szegedy et al,  Going Deeper with Convolutions, CVPR 2015, Free Online Version
    15. K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. CVPR 2016, Free Online Version
    16. V. Dumoulin, F. Visin, A guide to convolution arithmetic for deep learning, Arxiv
    17. S. Ioffe, C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, ICML 2013,  Arxiv
    18. M.D. Zeiler and R. Fergus, Visualizing and Understanding Convolutional Networks, ICML 2013, Arxiv
    19. G.E. Hinton, R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks.Science 313.5786 (2006): 504-507, Free Online Version
    20. G.E. Hinton, R. R. Salakhutdinov. Deep Boltzmann Machines. AISTATS 2009, Free online version.
    21. R. R. Salakhutdinov. Learning Deep Generative Models, Annual Review of Statistics and Its Application, 2015, Free Online Version
    22. Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on, Vol. 35(8) (2013): 1798-1828, Arxiv.
    23. G. Alain, Y. Bengio. What Regularized Auto-Encoders Learn from the Data-Generating Distribution, JMLR, 2014.
    24. Y. Bengio, P. Simard and P. Frasconi, Learning long-term dependencies with gradient descent is difficult. TNN, 1994, Free Online Version
    25. S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Computation, 1997, Free Online Version
    26. K. Greff et al, LSTM: A Search Space Odyssey, TNNLS 2016, Arxiv
    27. C. Kyunghyun et al, Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation, EMNLP 2014, Arxiv
    28. N. Srivastava et al, Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JLMR 2014
    29. Bahdanau et al, Neural machine translation by jointly learning to align and translate, ICLR 2015, Arxiv
    30. Xu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention, ICML 2015, Arxiv
    31. Koutník et al, A Clockwork RNN, ICML 2014, Arxiv
    32. Krueger, Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activation, ICLR 2018, Arxiv
    33. Sukhbaatar et al, End-to-end Memory Networks, NIPS 2015, Arxiv
    34. A. Graves et al, Neural Turing Machines, Arxiv
    35. A.Graves, Adaptive Computation Time for Recurrent Neural Networks, Arxiv
    36. A. Vaswan et al, Attention Is All You Need, NIPS 2017, Arxiv
    37. A. van der Oord et al., Pixel Recurrent Neural Networks, 2016, Arxiv
    38. C. Doersch, A Tutorial on Variational Autoencoders, 2016, Arxiv
    39. Ian Goodfellow, NIPS 2016 Tutorial: Generative Adversarial Networks, 2016, Arxiv
    40. Arjovsky et al, Wasserstein GAN, 2017, Arxiv
    41. T. White, Sampling Generative Network, NIPS 2016, Arxiv