We discuss problems related to identification/prediction of behavioral characteristics of an intelligent agent (e.g. a human, a robot, an avatar) based on noisy video footage. The attributes of interest include: patterns of motion (e.g. pose and position ), intentions, mood swings etc. Some of the attributes (e.g. intentions) are not directly observable, and need to be inferred from the other attributes. Others, such as pose and location, are partially observable but the observations are corrupted by noise.

Our general approach to the problem is Bayesian. More specifically, the hierarchical dynamics of the agent behavior is modeled by a telescoping/ layered Markov chain (TMC) which iteratively conditions the distribution of circumstantial attributes on the values taken by more basic ones. For instance, a person's pose is a random field taking values in a set of possible poses. The distribution with which the agent takes a particular pose depends on the agent's intentions, which can be modeled by components of the TMC at another level in the hierarchy.

Our approach to identification is based on nonlinear filtering type algorithms and optimal change-point detection for partially observable TMCs. Generally speaking, nonlinear filtering for partially observable TMCs is a particular case of the Hidden Markov Model (HMM) for vector-valued Markov chains. One of the main obstacles to efficient performance HMMs is the curse of dimensionality. To some degree, these problems could be mollified by introduction of particle filters, Rao-Blackwellization, and other methods, but the high dimensionality still remains to be a serious problem. Introduction of TMCs is just another step in the quest for reduction of computational complexity of HMMs.

Applications of TMC-based nonlinear filtering to analysis of video footage, presented in the paper, demonstrates practical potential of the approach.