Context. Interactive TV has not reached yet its full potential. How to make the use of interactivity in television content viable and attractive is something in evolution that can be seen with the popularization of new approaches as the use of second screen as interactive platform. Objective. This study aims at surveying existing research on Multiple Contents TV Synchronization in order to synthesize their results, classify works with common points, and identify needs for future research. Method. This paper reports the results of a systematic literature review and mapping study on TV Multiple Contents Synchronization published until middle 2013. As result, a set of 68 papers was generated and analyzed considering general information such as sources and time of publication; covered research topics; and synchronization aspects such as methods, channels, and precision. Results. Based on the obtained data, the paper provides a high level overview of the analyzed works; a detailed exploration of each used and proposed technique and its applications; and a discussion and proposal of a scenario overview and classification scheme based on the extracted data. 1. Introduction Multimedia systems allow the data streams integration of different types, including continuous (audio and video) and discrete media (text, data, and images). Synchronization is essential for the integration of these media [1] and is focus of researches for a long time [2, 3]. Most of these works use a common taxonomy proposed by Cesar and Chorianopoulos [4] to classify multimedia synchronization. This classification [5] is based on multimedia abstraction layers (Figure 1): in the media layer an application operates on a single continuous media stream, which is treated as a sequence of LDUs/MDUs (logical Data Units/Media Data Units); the stream layer allows the application to operate on continuous media streams as well as on groups of media streams; the object layer allows for simpler and exact specification of playout sequences, where each media object relates to a time axis and defines a sequence of events. Figure 1: Synchronization abstraction layers—[ 5]. In the media layer, the intrastream synchronization deals with the maintenance, during the playout, of the temporal relationship within each time-dependent media stream, that is, between the MDUs of the same stream. In the stream layer the interstream synchronization refers to the synchronization, during the playout, of the playout processes of different media streams involved in the application and live synchronization deals with the
References
[1]
P. Cesar and D. Geerts, “Past, present, and future of social TV: a categorization,” in Proceedings of the IEEE Consumer Communications and Networking Conference (CCNC '11), pp. 347–351, January 2011.
[2]
R. Malhotra, “Hybrid broadcast broadband TV: the way forward for connected TVs,” IEEE Consumer Electronics Magazine, vol. 2, no. 3, pp. 10–16, 2013.
[3]
A. Baba, K. Matsumura, S. Mitsuya et al., “Seamless, synchronous, and supportive: welcome to hybridcast: an advanced hybrid broadcast and broadband system,” IEEE Consumer Electronics Magazine, vol. 1, no. 2, pp. 43–52, 2012.
[4]
P. Cesar and K. Chorianopoulos, “The evolution of TV systems, content, and users toward interactivity,” Foundations and Trends in Human-Computer Interaction, vol. 2, no. 4, pp. 279–373, 2008.
[5]
T. Meyer, W. Effelsberg, and R. Steinmetz, “A taxonomy on multimedia synchronization,” in Proceedings of the 4th Workshop on Future Trends of Distributed Computing Systems, pp. 97–103, 1993.
[6]
H.-P. Dommel and S. K. Verma, “Multipoint synchronization protocol,” in Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC '04), vol. 5, pp. 4631–4636, October 2004.
[7]
Y. Ishibashi, A. Tsuji, and S. Tasaka, “Group synchronization mechanism for stored media in multicast communications,” in Proceedings of the 16th IEEE Annual Conference on Computer Communications (INFOCOM '97), vol. 2, pp. 692–700, April 1997.
[8]
R. Steinmetz, “Multimedia synchronization techniques: experiences based on different system structures,” in Proceedings of the 4th IEEE ComSoc International Workshop on Multimedia Communications, pp. 306–314, 1992.
[9]
G. Blakowski and R. Steinmetz, “A media synchronization survey: reference model, specification, and case studies,” IEEE Journal on Selected Areas in Communications, vol. 14, no. 1, pp. 5–35, 1996.
[10]
L. F. G. Soares, R. M. R. Costa, M. F. Moreno, and M. F. Moreno, “Multiple exhibition devices in DTV systems,” in Proceedings of the 17th ACM International Conference on Multimedia (MM '09), pp. 281–289, ACM, October 2009.
[11]
S. Straus, Evidence-Based Medicine: How to Practice and Teach EBM, Elsevier/Churchill Livingstone, 2005.
[12]
T. Dyb？, T. Dings？yr, and G. K. Hanssen, “Applying systematic reviews to diverse study types: an experience report,” in Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM '07), pp. 225–234, September 2007.
[13]
M. N. Islam, “A systematic literature review of semiotics perception in user interfaces,” Journal of Systems and Information Technology, vol. 15, no. 1, pp. 45–77, 2013.
[14]
G. Hachey and D. Ga?evi?, “Semantic web user interfaces: a systematic mapping study and review,” Submitted to Semantic Web—Interoperability, Usability, Applicability an IOS Press Journal.
[15]
H. M. Cooper, “Organizing knowledge syntheses: a taxonomy of literature reviews,” Knowledge in Society, vol. 1, no. 1, pp. 104–126, 1988.
[16]
F. Boronat, J. Lloret, and M. García, “Multimedia group and inter-stream synchronization techniques: a comparative study,” Information Systems, vol. 34, no. 1, pp. 108–131, 2009.
[17]
P. Hoschka, “An introduction to the synchronized multimedia integration language,” IEEE Multimedia, vol. 5, no. 4, pp. 84–88, 1998.
[18]
R. Vatavu, “There's a world outside your TV: exploring interactions beyond the physical TV screen,” in Proceedings of the 11th european conference on Interactive TV and video (EuroITV '13), pp. 143–152, 2013.
[19]
S. B. Kim, Y. H. Lee, J. H. Kim, and S. I. Kim, “A full digital video eeg system,” Engineering in Medicine and Biology, vol. 1, 1999.
[20]
S. Zeng, J. R. Powers, and H. Hsiao, “A new video-synchronized multichannel biomedical data acquisition system,” IEEE Transactions on Biomedical Engineering, vol. 47, no. 3, pp. 412–419, 2000.
[21]
N. Nitta, N. Babaguchi, and T. Kitahashi, “Automated annotation of significant scenes based on structure of broadcast sports video,” Electronics and Communications in Japan, vol. 86, no. 12, pp. 69–78, 2003.
[22]
J. Fawcett, B. Beyer, D. Hum, A. Ault, J. Krogmeier, and C. Taskiran, “Rich immersive sports experience: a hybrid multimedia system for content consumption,” in Proceedings of the 6th IEEE Consumer Communications and Networking Conference (CCNC '09), pp. 1–5, January 2009.
[23]
E. Marilly, A. Germaneau, B. Boidart, and O. Martinot, “Active TV: enter into a cross media, community-based experience,” in Proceedings of the 6th IEEE Consumer Communications and Networking Conference (CCNC '09), January 2009.
[24]
C. S. Oliveira, E. B. Dos Santos Jr., and H. C. de Oliveira, “Synchronizing Brazilian Sign Language (LIBRAS) on videos for open digital TV with educational purposes,” in Proceedings of the European Conference for the Applied Mathematics and Informatics, pp. 18–23, December 2010.
[25]
D. Y. Wohn and E.-K. Na, “Tweeting about TV: sharing television viewing experiences via social media message streams,” First Monday, vol. 16, no. 3, 2011.
[26]
L. B. Chen and M. Prokopi, “Resource-aware pairing device framework for ubiquitous cloud applications,” in Proceedings of the International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing,, pp. 252–258, 2012.
[27]
C. Cho, S. S. Kim, and K. D. Moon, “Enhancing home entertainment experience with virtual UPnP media renderers,” in Proceedings of the International Conference on Information and Communication Technology Convergence, pp. 270–271, 2012.
[28]
I. Nor？es, V. Furtado, and E. Furtado, “Maps-TV: a software architecture to support development of collaborative maps in DTVi,” in Proceedings of the 10th European conference on Interactive tv and video (EuroITV '12), pp. 165–173, 2012.
[29]
M. Maritan, “Um Estudo sobre o Uso de Marca？？es Autocontidas em Vídeo para Promo？？o de Intera？？o Sensível ao Conteúdo em Múltiplos Contextos,” in Proceedings of the 19th Brazilian Symposium on Multimedia and the Web (WebMedia '13), 2013.
[30]
D. S. Martins, L. S. Oliveira, and M. G. C. Da Pimentel, “Designing the user experience in iTV-based interactive learning objects,” in Berlin 28th ACM International Conference on Design of Communication (SIGDOC '10), pp. 243–250, September 2010.
[31]
J. Brunheroto, R. Chernock, P. Dettori et al., “Issues in data embedding and synchronization for digital television,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME '00), vol. 3, pp. 1233–1236, August 2000.
[32]
S. M. Park, Y. J. Lee, J. H. Choi, and J. S. Choi, “The design and implementation of data server for data broadcast service,” in Proceedings of the 10th IEEE International Conference on Electronics, Circuits and Systems (ICECS '03), pp. 1176–1179, December 2003.
[33]
M. Ma, D. Wilkes-Gibbs, and A. Kaplan, “IDTV broadcast applications for a handheld device,” in Proceedings of the IEEE International Conference on Communications, vol. 1, pp. 85–89, June 2004.
[34]
L. Zhang, J.-Z. Cui, and J.-X. Zhang, “A method for storage and transport of embedded rich media application,” in Proceedings of the 1st International Symposium on Computer Network and Multimedia Technology (CNMT '09), pp. 1–4, December 2009.
[35]
L. B. Yuste and H. Melvin, “Enhanced IPTV services through time synchronisation,” in Proceedings of the 14th IEEE International Symposium on Consumer Electronics (ISCE '10), pp. 1–6, June 2010.
[36]
I. Vaishnavi, P. Cesar, D. Bulterman, O. Friedrich, S. Gunkel, and D. Geerts, “From IPTV to synchronous shared experiences challenges in design: distributed media synchronization,” Signal Processing, vol. 26, no. 7, pp. 370–377, 2011.
[37]
B. Cheng, “Virtual browser for enabling multi-device web applications,” in Proceedings of the Workshop on Multi-device App Middleware (Multi-Device '12), pp. 1–6, 2012.
[38]
N. Q. K. Duong, C. Howson, and Y. Legallais, “Fast second screen TV synchronization combining audio fingerprint technique and generalized cross correlation,” in International Conference on Consumer Electronics, p. 4, 2012.
[39]
S. Soursos and N. Doulamis, “Connected TV and beyond,” in Proceedings of the Consumer Communications and Networking Conference, pp. 582–586, 2012.
[40]
E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns: Elements of Reusable Object-Oriented Software, Addison Wesley, 1994.
[41]
A. López, M. Noè, and G. Fernàndez, “Linkage of additional contents to moving objects and video shots in a generic media framework for interactive television,” in The International Society for Optical Engineering, vol. 5600 of Proceedings of SPIE, pp. 31–39, 2004.
[42]
T. A. Tavares, A. Burlamaqui, D. A. Da Silva Albino et al., “Sharing virtual acoustic spaces over interactive TV programs—presenting “virtual cheering” application,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME '04), vol. 3, pp. 2175–2178, June 2004.
[43]
M. Fink, M. Covell, and S. Baluja, “Social-and interactive-television applications based on real-time ambient-audio identification,” in Proceedings of the 11th edition of the European Interactive TV Conference series (EuroITV '06), p. 10, 2006.
[44]
R. Jaeger, P. Dunker, and A. Hanusch, “Enabling object based in-content placement for digital video television systems,” in Proceedings of the 8th IEEE International Symposium on Multimedia (ISM '06), vol. 2, pp. 662–666, December 2006.
[45]
E. Kawai, S. Yamaguchi, T. Akafuji, and K. Kandori, “A framework of a content distribution service synchronized with TV broadcasting programs,” in Proceedings of the 3rd IEEE Consumer Communications and Networking Conference (CCNC '06), vol. 2, pp. 1063–1067, January 2006.
[46]
J. Cha, Y. Seo, Y. Kim, and J. Ryu, “An authoring/editing framework for haptic broadcasting: passive haptic interactions using MPEG-4 BIFS,” in Proceedings of the 2nd Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, World Haptics (WHC '07), pp. 274–279, March 2007.
[47]
F. P. Margalef, A. López, and G. Fernández, “Frame-accuracy synchronisation for mobile TV interactivity,” in Proceedings of the IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB '09), pp. 1–5, May 2009.
[48]
T. W. L. Lai-Yeung and S. W. W. So, “TV as a multimedia synchronous communication for cooking and eating activities: analysis of TV cooking shows in Hong Kong,” in Proceedings of the IEEE International Symposium on Multimedia (ISM '10), pp. 302–307, December 2010.
[49]
L. B. Yuste, S. M. Saleh Al-Majeed, H. Melvin, and M. Fleury, “Effective synchronisation of hybrid broadcast and broadband TV,” in Proceedings of the IEEE International Conference on Consumer Electronics (ICCE '12), pp. 160–161, January 2012.
[50]
G. Forbes, “Closed captioning transmission and display in digital television,” in Proceedings of the International Workshop on Digital and Computational Video, pp. 126–131, 2001.
[51]
J. Sung, D. Kim, Y. Doh, H. Lee, and J. S. Choi, “Broadcasting associated service architecture and its synchronization support,” in Proceedings of the IEEE International Symposium on Consumer Electronics, pp. 251–256, September 2004.
[52]
F. Hendrickx, T. Beckers, N. Oorts, and R. D. van Walle, “An integrated approach for device independent publication of complex multimedia documents,” in Proceedings of the 9th IASTED International Conference on Internet and Multimedia Systems and Applications (IMSA '05), pp. 347–352, August 2006.
[53]
M. C. Marques Neto and C. A. S. Santos, “An event-based model for interactive live TV shows,” in Proceedings of the 16th ACM International Conference on Multimedia (MM '08), pp. 845–848, October 2008.
[54]
J. Yun, H. Lee, and K. Park, “Orchestral media: the method for synchronizing single media with multiple devices for ubiquitous home media services,” in Proceedings of the 3rd International Conference on Convergence and Hybrid Information Technology (ICCIT '08), vol. 1, pp. 340–345, November 2008.
[55]
M. C. Maques Neto and C. A. S. Santos, “An approach based on events for treating the late tuning problem in interactive live TV shows,” in Proceedings of the ACM International workshop on Events in Multimedia, pp. 41–48, 2009.
[56]
J. Yun, J. Jang, and K. Park, “Real-sense media synchronization technology based on the SMMD,” in Proceedings of the International Conference on Consumer Electronics (ICCE '10), pp. 71–72, January 2010.
[57]
L. Lancerica, L. Dairaine, F. de Belleville, H. Thalmensy, and C. Fraboul, “MITv—a solution for an interactive TV based on IP multicast over satellite,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME '04), vol. 3, pp. 2159–2162, June 2004.
[58]
E. Papaioannou, K. Karpouzis, P. de Cuetos et al., “Melisa—a distributed multimedia system for multi-platform interactive sports content broadcasting,” in Proceedings of the 30th EUROMICRO Conference, vol. 30, pp. 222–229, September 2004.
[59]
P. Leroux, V. Verstraete, F. de Turck, and P. Demeester, “Efficient management of synchronised interactive services through the design of MCDP middleware,” in Proceedings of the Australasian Telecommunication Networks and Applications Conference (ATNAC '07), pp. 215–220, December 2007.
[60]
M. F. Moreno, R. M. D. R. Costa, and L. F. G. Soares, “Sincronismo entre fluxos de midia continua e aplicacoes multimidia em redes por difusao,” in Proceedings of the 14th Brazilian Symposium on Multimedia and the Web (WebMedia '08), pp. 202–209, 2008.
[61]
R. M. D. R. Costa, M. F. Moreno, and L. F. G. Soares, “Ginga-NCL: supporting multiple devices,” in Proceedings of the 15th Brazilian Symposium on Multimedia and the Web (WebMedia '09), October 2009.
[62]
H. Lee, C. Yoon, and H.-W. Lee, “Smart Screen Service platform enabling target advertising,” in Proceedings of the International Conference on Information and Communication Technology Convergence (ICTC '10), pp. 527–528, November 2010.
[63]
K. Matsumura, M. J. Evans, Y. Shishikui, and A. McParland, “Personalization of broadcast programs using synchronized internet content,” in Proceedings of the International Conference on Consumer Electronics (ICCE '10), pp. 145–146, January 2010.
[64]
C. Howson, E. Gautier, P. Gilberton, A. Laurent, and Y. Legallais, “Second screen TV synchronization,” in Proceedings of the 1st IEEE International Conference on Consumer Electronics (ICCE '11), pp. 361–365, Berlin, Germany, September 2011.
[65]
J. Lyle, S. Monteleone, S. Faily, D. Patti, and F. Ricciato, “Cross-platform access control for mobile web applications,” in Proceedings of the IEEE International Symposium on Policies for Distributed Systems and Networks, pp. 37–44, 2012.
[66]
F. M. Matsubara and M. Kawamori, “Lightweight interactive multimedia environment for TV,” IEEE Transactions on Consumer Electronics, vol. 57, no. 1, pp. 283–287, 2011.
[67]
A. Baba and K. Matsumura, “Seamless, synchronous, and supportive: welcome to hybridcast: an advanced hybrid broadcast and broadband system,” Consumer Electronics Magazine, vol. 1, no. 2, pp. 43–52, 2012.
[68]
C. Kohnen, C. Kobel, and N. Hellhund, “A DVB/IP streaming testbed for hybrid digital media content synchronization,” in Proceedings of the International Conference on Consumer Electronics, pp. 136–140, 2012.
[69]
L. B. Yuste and H. Melvin, “Client-side multisource media streams multiplexing for HbbTV,” in Proceedings of the International Conference on Consumer Electronics, pp. 1–5, 2012.
[70]
A. L. S. Gradvohl and Y. Iano, “An approach for interactive television based on insertion of hypermedia information in MPEG standard video,” in Proceedings of the IEEE International Symposium on Consumer Electronics, pp. 25–30, September 2004.
[71]
H. M. Stokking, M. O. van Deventer, O. a. Niamut, F. a. Walraven, and R. N. Mekuria, “IPTV inter-destination synchronization: a network-based approach,” in Proceedings of the International Conference on Intelligence in Next Generation Networks, pp. 1–6, 2010.
[72]
F. Boronat, M. Montagud, H. Stokking, and O. Niamut, “The need for inter-destination synchronization for emerging social interactive multimedia applications,” Communications Magazine, vol. 50, no. 11, pp. 150–158, 2012.
[73]
M. Montagud and F. Boronat, “Enhanced adaptive RTCP-based Inter-destination multimedia synchronization approach for distributed applications,” Computer Networks, vol. 56, no. 12, pp. 2912–2933, 2012.
[74]
M. Montagud, F. Boronat, H. Stokking, and R. Brandenburg, “Inter-destination multimedia synchronization: schemes, use cases and standardization,” Multimedia Systems, vol. 18, no. 6, pp. 459–482, 2012.
[75]
B. Li, J. H. Errico, H. Pan, and I. Sezan, “Bridging the semantic gap in sports video retrieval and summarization,” Journal of Visual Communication and Image Representation, vol. 15, no. 3, pp. 393–424, 2004.
[76]
R. G. Cattelan, C. Teixeira, R. Goularte, and M. D. G. C. Pimentel, “Watch-and-comment as a paradigm toward ubiquitous interactive video editing,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 4, no. 4, article 28, 2008.
[77]
A. Crivellaro and M. P. de Almeida, “The real time audience system, one minute or nothing,” in Proceedings of the 4th International Conference on Automated Solutions for Cross Media Content and Multi-Channel Distribution,, pp. 204–207, November 2008.
[78]
J. E. Garcia, A. Ortega, E. Lleida, T. Lozano, E. Bernues, and D. Sanchez, “Audio and text synchronization for TV news subtitling based on automatic speech recognition,” in Proceedings of the IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB '09), May 2009.
[79]
V. G. Motti, R. Fagá, R. G. Catellan, M. Da Gra？a C. Pimentel, and C. A. C. Teixeira, “Collaborative synchronous video annotation via the watch-and-comment paradigm,” in Proceedings of the 7th European Conference on European Interactive Television Conference (EuroITV '09), pp. 67–76, June 2009.
[80]
J. Gao, Q. Zhao, and Y. Yan, “Automatic synchronization of live speech and its transcripts based on a frame-synchronous likelihood ratio test,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '10), pp. 1622–1625, March 2010.
[81]
M. de Castro, D. Carrero, L. Puente, and B. Ruiz, “Real-time subtitle synchronization in live television programs,” in Proceedings of the IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB '11), June 2011.
[82]
D. Das, P. Sinha, A. Ghose, and C. Bhaumik, “An interactive system using digital broadcasting and quick response code,” in Proceedings of the 15th IEEE International Symposium on Consumer Electronics (ISCE '11), pp. 397–400, June 2011.
[83]
F. L. S. Ferreira, F. H. Lemos, G. P. B. Neto, T. M. U. de Araujo, and G. L. D. S. Filho, “Providing support for sign languages in middlewares compliant with ITU J.202,” in Proceedings of the 13th IEEE International Symposium on Multimedia (ISM '11), pp. 149–156, December 2011.
[84]
I. Bisio, A. Delfino, G. Luzzati et al., “Opportunistic estimation of television audience through smartphones,” in Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems, pp. 1–5, 2012.
[85]
S.-H. Dai, Z.-Y. Zhu, and G.-Y. Lin, “Synchronization model design and implementation of distributed multimedia system,” in Proceedings of the 3rd International Conference on Innovative Computing Information and Control (ICICIC '08), p. 209, June 2008.
[86]
H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “RTP: a transport protocol for real-time applications,” RFC 3550, 2003.
[87]
G. Harboe, N. Massey, C. Metcalf, D. Wheatley, and G. Romano, “The uses of social television,” Computers in Entertainment, vol. 6, no. 1, article 8, 2008.
