Natural language description of human activities from video images based on concept hierarc.pdf

Natural Language Description of Human Activites from Video Images Based on Concept Hierarchy of Actions Atsuhiro Kojima and Takeshi Tamura Library and Science Information Center, Osaka Prefecture University 1-1 Gakuen-cho, Sakai, Osaka 599-8531, JAPAN Kunio Fukunaga Graduate School of Engineering, Osaka Prefecture University 1-1 Gakuen-cho, Sakai, Osaka 599-8531, JAPAN Abstract. We propose a method of describing human activities from video images based on concept hierarchies of actions. Major difficulty in transforming video images into textual descriptions is how to bridge a semantic gap between them, which is also known as inverse Hollywood problem. In general, the concepts of events or actions of human can be classified by semantic primitives. By making correspondence between these concepts and the semantic features extracted from video images, appropriate syntactic components such as verbs, objects, etc. are determined and then translated into natural language sentences. We also demonstrate the performance of the proposed method by several experiments. Keywords: natural language generation, concept hierarchy, semantic primitive, position/posture estimation of human, case frame 1. Introduction It is becoming popular to introduce natural language concepts into a vision system. Traffic surveillance system, for instance, represents moving vehicles by series of verbs or short sentences in place of numerical expressions of the objects’ location(Kollnig et al., 1994; Nagel, 1994). In their method, the motion of vehicles are estimated from displacement vector field and associ- ated with motion verbs by evaluating attributes of each trajectory segment, such as vehicle speed. Comparing with vehicle movements, description of human behavior in image sequences is more complicated. In a visual surveil- lance system(Thonnat and Rota, 1999), human behavior is represented by scenarios, i.e. predefined sequences of events. The scenario is then translated into text by filling a te