San José State University
Department of Computer Science
CS-180H: Individual Studies for Honors

Instructor

Robert Bruce

Prerequisites

At least junior standing as computer science major. GPA of 3.5 or higher in the major and department chair consent.

Course Description

Senior project on advanced topics in computer science as determined by the instructor. Written paper or oral presentation of the project required. Intended for students graduating with departmental honors.

Learning Outcomes

Upon successful completion of this course, students will be able to:

  • Use the Active Appearance Modelling (AAM) algorithm from OpenFace to locate and project 2D facial features onto a 3D animated avatar in Maya in real-time.
  • Use the C++ Maya plugin Application Programmer’s Interface (API) for direct manipulation and access to 3D computer graphics model models in Maya.
  • Use shared memory and semaphores for synchronized interprocess communication (i.e. avoiding race conditions) between OpenFace and the Maya API.
  • Creating a real-time markerless motion capture system.
  • Apply software engineering principles through the entire development lifecycle including design, implementation, testing, release, and maintenance.

Required Texts/Readings

Complete Maya Programming (Volume I) by David A. D. Gould. ISBN: 978-1558608351.

Complete Maya Programming (Volume II): An In-depth Guide to 3D Fundamentals, Geometry, and Modeling by David Gould. ISBN: 978-0120884827.

Advanced programming in the UNIX environment by W. Richard Stevens and Stephen A. Rago.

Grading

In this course the student will develop a markerless motion capture system to project facial expressions onto a virtual 3D animated avatar in real-time. This project requires knowledge of systems programming, 2D and 3D computer graphics, Maya animation/modelling software, and computer vision.

Deliverables

Foundation
OpenFace / Maya relationship
  1. Establish shared memory
  2. OpenFace dumps 68 data points to shared memory
    Interprocess Communication (IPC)
  3. Maya plug-in (IPC)
Stage 2
Rigging in Maya
  1. Develop fully rigged model
  2. Develop expression-driven rigs
    (Cheeks move when mouth corners move)
Stage 3
Maya plugin core
  1. Manipulating bones using plugin
  2. Correcting for head roll to get true horizontal/vertical delta distances
  3. Apply delta distances to bones
Stage 4
User Interface
  1. Start/stop bone manipulation
  2. Choose bones to be manipulated/mapped, assign OpenFace data to map to which bone

Grading Policy

This class is graded credit / no credit.

Grading Rubric

Deliverable 10 points 18 points 25 points
Foundation
  • Establishes shared memory.
  • Writes 68 data points into memory.
  • Establishes shared memory.
  • Locks shared memory using mutex or semaphore.
  • Writes 68 data points into shared memory.
  • Frees memory lock using mutex or semaphore after writing 68 data points.
  • Establishes shared memory.
  • Locks shared memory using mutex or semaphore.
  • Writes 68 data points into shared memory.
  • Frees memory lock using mutex or semaphore after writing 68 data points.
Stage 2
  • Develops partially rigged model to manipulate skull in yaw, pitch, and roll.
  • Develops fully rigged model to manipulate skull (yaw, pitch, roll), eyes (pupils), eyebrows, nose, mouth (lips and jaw).
  • Develops fully rigged model to manipulate skull (yaw, pitch, roll), eyes (pupils), eyebrows, nose, mouth (lips and jaw).
  • Develops expression-driven rigs for left and right cheeks.
Stage 3
  • Compute delta distance for yaw, pitch, roll from previous to current frame.
  • Compute delta distances for 68-data points from previous to current frame.
  • Apply delta distance for yaw, pitch, and roll to skull rig.
  • Apply delta distance from 68 data points in OpenFace to eyebrows, eyelids, and eyes.
  • Apply delta distance for yaw, pitch, and roll to skull rig.
  • Apply delta distances from 68 data points in OpenFace to eyebrows, eyelids, eyes, upper lip, lower lip, jaw.
  • Drive left and right cheeks from delta distances applied to mouth corners (expression-driven rig).
Stage 4
  • Add UI control buttons to start/stop bone manipulation. Buttons have no functionality yet.
  • Add UI control to record (start/stop) delta distances applied to rig for skull (yaw, pitch, roll).
  • Add UI control to record (start/stop) delta distances applied to rig for skull (yaw, pitch, roll), eyebrows, eyes, upper/lower mouth, jaw, and cheeks.

Total possible points for this project is 100. To receive credit for this class one needs to score 70 points or above.

Course Schedule

Week Topics, Readings, Assignments, Deadlines
1 Overview for building a real-time motion capture pipeline
2 Shared memory, Semaphores, and Inter-process communication
3 Introduction to Pthreads (multi-threading for speed)
4 Mapping 2D shapes onto a 3D avatar
5 Introduction (overview) of the Maya C++ API
6 Dissection of a boilerplate (minimal code) for a Maya C++ plugin.
7 Mesh manipulation (vertices) in the Maya C++ API
8 Introduction to rigging in Maya
9 Lip segmentation and tracking - rigging tips and tricks.
10 Mesh deformation C++ API: Manipulating vertices in Maya.
11 Writing multi-threaded C++ Maya plugins
12 Developing Maya user interfaces plugins in QT.
13 Maya plugin File I/O.
14 Introduction to Directed Acyclic Graphs (DAG) for Maya scene descriptions.
15 Project demonstration.
DUE: Final Project.