We present an integrated framework on personalized sports video customization, which addresses three research issues: semantic video annotation, personalized video retrieval and summarization, and system adaptation. Sports video annotation serves as the foundation of the video customization system. To acquire detailed description of video content, external web text is adopted to align with the related sports video according to their semantic correspondence. Based on the derived semantic annotation, a user-participant multiconstraint 0/1 Knapsack model is designed to model the personalized video customization, which can unify both video retrieval and summarization with different fusion parameters. As a measure to make the system adaptive to the particular user, a social network based system adaptation algorithm is proposed to learn latent user preference implicitly. Both quantitative and qualitative experiments conducted on twelve broadcast basketball and football videos validate the effectiveness of the proposed method. 1. Introduction The proliferation of advanced program production technology and multiple TV broadcast channels have contributed to an amazing growth of sports video content and its increasing popularity among the public. However, such increasing availability has not yet been accompanied by an improvement in its accessibility, which means that audiences can do nothing but passively watch the whole match edited by studio professionals once they choose it. Since interesting segments usually account for a small portion of the whole match; such passive watching mode not only impairs audiences' viewing experience but also wastes their time and money. To solve this problem, the ability to provide personalized video content in accordance to the features of individual viewers is of great importance. Intuitively, viewers difference first comes through in their diverse preference towards semantic content where particular players and events are appearing in the video. For example, a Beckham's fan may mainly focus his attention on this football star than any other players, while an NBA's audience may prefer to watch the slam dunk than any other events. To meet these requirements, the source video has to be analyzed in a more refined scale and higher semantic level. More precisely, the video analysis should not merely tag some salient events with simple concepts, for example, shots in basketball or fouls in football, but annotate various events with detailed semantic description, including the involved player(s), event type(s), and result consequence.
References
[1]
Y. Rui, A. Gupta, and A. Acero, “Automatically extracting highlights for TV baseball programs,” in Proceedings of the ACM International Multimedia Conference and Exhibition, pp. 105–115, Los Angeles, Calif, USA, 2000.
[2]
M. Xu, N. C. Maddage, C. S. Xu, M. S. Kakanhalli, and Q. Tian, “Creating audio keywords for event detection in soccer video,” in Proceeding of the IEEE International Conference on Multimedia and Expo (ICME '03), vol. 2, pp. 281–284, Baltimore, Md, USA, 2003.
[3]
Z. Xiong, R. Radhakrishnan, A. Divakaran, and T. S. Huang, “Audio events detection based highlight extraction from baseball, golf and soccer games in a United Framework,” in Proceeding of the IEEE International Conference on Acuoustics, Speech and Signal Processing (ICASSP '03), vol. 5, pp. 632–635, Hong Kong, April 2003.
[4]
L.-Y. Duan, M. Xu, and Q. Tian, “Semantic shot classification in sports video,” in Storage and Retrieval for Media Database, vol. 5021 of Proceedings of SPIE, pp. 300–313, January 2003.
[5]
Y.-P. Tan, D. D. Saur, S. R. Kulkarni, and P. J. Ramadge, “Rapid estimation of camera motion from compressed video with application to video annotation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 10, no. 1, pp. 133–146, 2000.
[6]
D. Zhang and S.-F. Chang, “Event detection in baseball video using superimposed caption recognition,” in Proceedings of the ACM International Multimedia Conference and Exhibition, pp. 315–318, Juan-les-Pins, France, December 2002.
[7]
J. Assfalg, M. Bertini, C. Colombo, A. Del Bimbo, and W. Nunziati, “Semantic annotation of soccer videos: automatic highlights identification,” Computer Vision and Image Understanding, vol. 92, no. 2-3, pp. 285–305, 2003.
[8]
A. Ekin, A. M. Tekalp, and R. Mehrotra, “Automatic soccer video analysis and summarization,” IEEE Transactions on Image Processing, vol. 12, no. 7, pp. 796–807, 2003.
[9]
A. S. David and E. O. C. Eoel, “Event detection in Field sports video using audiovisual features and a support vector machine,” IEEE Transaction on Circuits and Systems for Video Technology, vol. 15, no. 10, pp. 1225–1233, 2005.
[10]
C.-L. Huang, H.-C. Shih, and C.-Y. Chao, “Semantic analysis of soccer video using dynamic Bayesian network,” IEEE Transactions on Multimedia, vol. 8, no. 4, pp. 749–760, 2006.
[11]
L. Xing, Q. Ye, W. Zhang, Q. Huang, and H. Yu, “A scheme for racquet sports video analysis with the combination of audio-visual information,” in Visual Communications and Image Processing, vol. 5960 of Proceedings of SPIE, pp. 259–267, 2005.
[12]
L.-Y. Duan, M. Xu, T.-S. Chua, Q. Tian, and C.-S. Xu, “A mid-level representation framework for semantic sports video analysis,” in Proceedings of the ACM International Multimedia Conference and Exhibition, pp. 33–44, Berkeley, Calif, USA, 2003.
[13]
M. Xu, L. Duan, C. Xu, M. Kankanhalli, and Q. Tian, “Event detection in basketball video using multi-modalities,” in Proceeding of IEEE Pacific Rim Conference on Multimedia (PCM '03), vol. 3, pp. 1526–1530, Singapore, December 2003.
[14]
K. Wan and C. Xu, “Efficient multimodal features for automatic soccer highlight generation,” in Proceedings of the International Conference on Pattern Recognition (ICPR '04), vol. 3, pp. 973–976, Cambridge, UK, August 2004.
[15]
M. H. Kolekar and S. Sengupta, “A hierarchical framework for generic sports video classification,” in Proceedings of the 7th Asian Conference on Computer Vision (ACCV '06), vol. 3852 of Lecture Notes in Computer Science, pp. 633–642, Springer, Hyderabad, India, January 2006.
[16]
M. Xu, L.-Y. Duan, C.-S. Xu, and Q. Tian, “A fusion scheme of visual and auditory modalities for event detection in sports video,” in Proceedings IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP '03), vol. 3, pp. 189–192, Hong Kong, 2003.
[17]
M. Han, W. Hua, W. Xu, and Y. Gong, “An integrated baseball digest system using maximum entropy method,” in Proceedings of the ACM International Multimedia Conference and Exhibition, pp. 347–350, Juan-les-Pins, France, December 2002.
[18]
M. H. Kolekar and S. Sengupta, “Event-importance based customized and automatic cricket highlight generation,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME '06), pp. 1617–1620, July 2006.
[19]
C. Jung and J. Kim, “Player information extraction for semantic annotation in golf videos,” IEEE Transactions on Broadcasting, vol. 55, no. 1, pp. 79–83, 2009.
[20]
N. Babaguchi, Y. Kawai, and T. Kitahashi, “Event based indexing of broadcasted sports video by intermodal collaboration,” IEEE Transactions on Multimedia, vol. 4, no. 1, pp. 68–75, 2002.
[21]
H. Xu and T.-S. Chua, “The fusion of audio-visual features and external knowledge for event detection in team sports video,” in Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval (MIR '04), pp. 127–134, New York, NY, USA, October 2004.
[22]
C. S. Xu, J. Wang, K. Wan, Y. Li, and L. Duan, “Live sports event detection based on broadcast video and web-casting text,” in Proceedings of the 14th Annual ACM International Conference on Multimedia (MM '06), pp. 221–230, Santa Barbara, Calif, USA, October 2006.
[23]
D. Tjondronegoro, Y.-P. P. Chen, and B. Pham, “Integrating highlights for more complete sports video summarization,” IEEE Multimedia, vol. 11, no. 4, pp. 22–37, 2004.
[24]
A. Ekin and A. M. Tekalp, “Generic play-break event detection summarization and hierarchical sports video analysis,” in Proceedings of the IEEE International Conference on Multimedia & Expo (ICME '03), vol. 1, pp. 167–172, Baltimore, Md, USA, July 2003.
[25]
M. Fleischman and D. Roy, “Situated models of meaning for sports video retrieval,” in Proceeding of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL '07), pp. 37–44, Rochester, NY, USA, April 2007.
[26]
N. Babaguchi, K. Ohara, and T. Ogura, “Learning personal preference from viewer's operations for browsing and its application to baseball video retrieval and summarization,” IEEE Transactions on Multimedia, vol. 9, no. 5, pp. 1016–1025, 2007.
[27]
W. Gao, Q.-M. Huang, S. Q. Jiang, and P. Zhang, “Sports video summarization and adaptation for application in mobile communication,” Journal of Zhejiang University: Science A, vol. 7, no. 5, pp. 819–829, 2006.
[28]
Y. Wei, S. M. Bhandarkar, and K. Li, “Video personalization in resource-constrained multimedia environments,” in Proceedings of the ACM International Multimedia Conference and Exhibition, pp. 902–911, Augsburg, Germany, September 2007.
[29]
B. L. Tseng, C. Y. Lin, and J. R. Smith, “Video summarization and personalization for pervasive mobile devices,” in Storage and Retrieval for Media Database, vol. 4676 of Proceedings of SPIE, pp. 359–370, 2002.
[30]
Y. Zhang, X. Zhang, C. Xu, and H. Lu, “Personalized retrieval of sports video,” in Proceedings of the International Workshop on Multimedia Information Retrieval (MIR '07), pp. 313–322, Augsburg, Germany, September 2007.
[31]
A. Amir, M. Berg, and H. Permuter, “Mutual relevance feedback for multimodal query formulation in video retrieval,” in Proceedings of International Workshop on Multimedia Information Retrieval (MIR '05), pp. 17–24, Singapore, November 2005.
[32]
T. Syeda-Mahmood and D. Ponceleon, “Learning video browsing behavior and its application in the generation of video previews,” in Proceedings of the 9th ACM International Conference on Multimedia (MM '01), pp. 119–128, Ottawa, Canada, September 2001.
[33]
J. Zimmerman, K. Kurapati, A. L. Buczak, D. Schaffer, S. Gutta, and J. Martino, “TV personalization system: design of a TV show recommender engine and interface,” in Personalized Digital Television: Targeting Programs to Individual Viewers, pp. 27–51, Kluwer Academic Publishers, Dordrecht, The Netherlands, 2004.
[34]
A. Jaimes, N. Sebe, and D. Gatica-Perez, “Human-centered computing: a multimedia perspective,” in Proceedings of the 14th Annual ACM International Conference on Multimedia (MM '06), pp. 855–864, Santa Barbara, Calif, USA, September 2006.
[35]
C. Xu, J. J. Wang, H. Q. Lu, and Y. F. Zhang, “A novel framework for semantic annotation and personalized retrieval of sports video,” IEEE Transactions on Multimedia, vol. 10, no. 3, pp. 421–436, 2008.
[36]
C. Liang, Y. Zhang, C. S. Xu, J. Q. Wang, and H. Q. Lu, “A hierarchical semanticsmatching approach for sports video annotation,” in Proceedings of the 10th Pacific Rim Conference on Multimedia (PCM '09), vol. 5879 of Lecture Notes in Computer Science, pp. 684–696, Bangkok, Thailand, December 2009.
[37]
L. Wang, M. Lew, and G. Xu, “Offense based temporal segmentation for event detection in soccer video,” in Proceedings of the 6th ACM SIGMM International Workshop on Multimedia Information Retrieval (MIR '04), pp. 259–266, New York, NY, USA, October 2004.
[38]
F. Dufaux and J. Konrad, “Efficient, robust, and fast global motion estimation for video coding,” IEEE Transactions on Image Processing, vol. 9, no. 3, pp. 497–501, 2000.
[39]
C. Xu, Y. F. Zhang, G. Y. Zhu, Y. Rui, H. Q. Lu, and Q. M. Huang, “Using webcast text for semantic event detection in broadcast sports video,” IEEE Transaction on Multimedia, vol. 10, no. 7, pp. 1342–1355, 2008.
[40]
S. B. Needleman and C. D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequence of two proteins,” Journal of Molecular Biology, vol. 48, no. 3, pp. 443–453, 1970.
[41]
N. Babaguchi, Y. Kawai, T. Ogura, and T. Kitahashi, “Personalized abstraction of broadcasted American football video by highlight selection,” IEEE Transactions on Multimedia, vol. 6, no. 4, pp. 575–586, 2004.
[42]
B. Merialdo, K. T. Lee, D. Luparello, and J. Roudaire, “Automatic construction of personalized TV News programs,” in Proceedings of the ACM International Multimedia Conference, pp. 323–331, Orlando, Fla, USA, October 1999.
[43]
J. Scott, Social Network Analysis: A Handbook, Sage, Newbury Park, Calif, USA, 1991.
[44]
C.-Y. Weng, W.-T. Chu, and J.-L. Wu, “RoleNet: movie analysis from the perspective of social networks,” IEEE Transactions on Multimedia, vol. 11, no. 2, pp. 256–271, 2009.
[45]
A. Vinciarelli and S. Favre, “Broadcast news story segmentation using social network analysis and hidden Markov models,” in Proceedings of the ACM International Multimedia Conference and Exhibition, pp. 261–264, Augsburg, Germany, September 2007.
[46]
L. He, E. Sanocki, A. Gupta, and J. Grudin, “Auto-summarization of audio-video presentations,” in Proceedings of the ACM International Multimedia Conference and Exhibition, pp. 489–498, Orlando, Fla, USA, October 1999.
[47]
R. Likert, “A technique for the measurement of attitudes,” Archives of Psychology, vol. 22, no. 140, pp. 1–55, 1932.
