9967 – Machine Learning-Based Video Analysis, Detection, and Prediction


The proposed invention represents techniques for the machine learning-based video analysis, detection, and prediction in various video processing contexts. The proposed system can be configured to provide role inference and action anticipation in the context of team sports.




Forecasting human actions directly from video data is a critical ability for modern vision-based intelligent systems. The invention provides a unified framework that includes an inference stage and an anticipation stage for forecasting actions in team sports, which captures one or more significant characteristics of human team activities, such as dynamic scenes, group interactions, and diverse human actions.

In an example inference stage, the team strategy and the role of a player, two hidden variables in a scene of a team sport, are inferred via a multi-class classifier and a Spatio-temporal Markov Random Field (MRF) model, respectively. In the anticipation stage, by integrating the inference results with visual observations, a multi-layer perceptron is configured for anticipating future actions of key players in an online fashion, allowing the anticipation output to evolve over time.

The method for determining the role of sports players in streaming videos comprises recognizing a semantic label that describes the team activity at each frame and identifying a role label for each player that indicates the expected responsibility of that player. Method for anticipating future actions of players comprises predicting a sub-group of players who are likely to interact with a ball or the like and therefore influence the play and anticipating the unseen future actions of players based on the inferred team activity label and role label.

Potential Applications

  • Analysis of team performances and discovery of team patterns
  • Video games that animate sports players
  • Camera control in sports photography

Contact Information

Name: Martin Teschl

Email: mt439@cornell.edu

Phone: (607) 254-4454