Today sports analysis systems draw the attention of many commercial entities and are providing many opportunities for computer vision researchers to study and develop automated sports analysis systems. The primary objective of a sports video analysis is to interpret low-level visual features to high-level semantics that can be understood by the end user. Although all sports video analysis systems are based on similar principles at the low-level visual extraction, interpreting this visual information to high-level human understanding is extremely specific to the sport in question. This project is the very first attempt to evaluate the jockeys’ performance from horse-race TV broadcasts. The aim of this thesis is to extract high-level information from the horse race by automatically detecting and tracking jockeys, specifically at turning points of the race. However, detecting and tracking of jockeys are extremely difficult. This is because in most horse-races there are more than six jockeys and they closely follow each other to gain a leading position, consequently locating jockeys and maintaining their identities through a video sequence is highly challenging due to the frequent occlusion. In addition, the background in horse-race videos is continually changing, thus there is not only background clutter but the jockeys themselves may be obscured by obstacles such as trees, towers and bars. To tackle these challenges, a context-aware analysis system is proposed. The proposed system is developed based on deterministic reasoning which was obtained by observing various horse races. One important property of horse-races is the group dynamic behaviour of jockeys in the race. The jockeys race around a circular track and the camera typically follow the jockeys, thus the jockeys relative to each other tend to move as a slowly changing group. This homogeneous characteristic of jockeys in the race is very useful, especially when local information of jockeys is poor or abrupt due to occlusion or background clutter. The proposed system combines multiple cues, such as local jockeys’ information, background characteristics and group dynamic properties, to detect and track jockeys around a turning segment of a horse-race. We demonstrate that our proposed system can be generalized to work on other domains. The outline of the proposed model contains three modules; (1) scene analysis (2) jockey detection and, (3) tracking and data association. The main objective of scene analysis is to extract the turning segment in the race. The detection of jockeys is accomplished by determining the location of each jockey’s cap. For the tracking of the jockeys, we implement a robust hierarchical tracking scheme which iteratively adapts and updates itself by gathering the jockeys’ properties at each of the point, object and group levels. To boost the tracking accuracy, data association is applied as a multi-target management system to maintain multiple jockeys’ identities over time, to initialise the tracking and to terminate trajectories.