Mental image directed semantic theory (MIDST) has proposed an omnisensory mental image model and its description language . This language is designed to represent and compute human intuitive knowledge of space and can provide multimedia expressions with intermediate semantic descriptions in predicate logic. It is hypothesized that such knowledge and semantic descriptions are controlled by human attention toward the world and therefore subjective to each human individual. This paper describes expression of human subjective knowledge of space and its application to aware computing in cross-media operation between linguistic and pictorial expressions as spatial language understanding. 1. Introduction The serious need for more human-friendly intelligent systems has been brought by rapid increase of aged societies, floods of multimedia information over the WWW, development of robots for practical use, and so on. For example, it is very difficult for people to exploit necessary information from the immense multimedia contents over the WWW. It is still more difficult to search for desirable contents by queries in different media, for example, text queries for pictorial contents. In this case, intelligent systems facilitating cross-media references are helpful and worth developing. In this research area so far, it has been most conventional that conceptual contents conveyed by information media such as languages and pictures are represented in computable forms independent of each other and translated via so-called “transfer” processes which are often ad hoc and very specific to task domains [1–3]. In order to systematize cross-media operation, however, it is needed to develop such a computable knowledge representation language for multimedia contents that should have at least a good capability of representing spatiotemporal events perceived by people in the real world. For this purpose, mental image directed semantic theory (MIDST) has proposed a model of human mental image and its description language (Language for mental-image description) [4]. This language is capable of formalizing human omnisensory mental images (equal to multimedia contents, here) in predicate logic, while other knowledge description schema [5, 6] are too coarse or linguistic (or English-like) to formalize them in an integrative way as intended here. is employed for many-sorted predicate logic and has been implemented on several versions of the intelligent system IMAGES [4, 7] and there is a feedback loop between them for their mutual refinement unlike other similar theories [8, 9]. As
References
[1]
A. Yamada, A. Yamada, H. Ikrda, et al., “Reconstructing spatial image from natural language texts,” in Proceedings of the 15th International Conference on Computational Linguistics (COLING '90), Nantes, France, 1992.
[2]
P. Olivier and J. Tsujii, “A computational view of the cognitive semantics of spatial expressions,” in Proceedings of the 32nd annual meeting on Association for Computational Linguistics (ACL '94), Las Cruces, New Mexico, 1994.
[3]
G. Adorni, M. Di Manzo, and F. Giunchiglia, “Natural language driven image generation,” in Proceedings of the 10th International Conference on Computational Linguistics (COLING '84), pp. 495–500, 1984.
[4]
M. Yokota and G. Capi, “Cross-media operations between text and picture based on mental image directed semantic theory,” WSEAS Transactions on Information Science and Applications, vol. 2, no. 10, pp. 1541–1550, 2005.
[5]
J. F. Sowa, Knowledge Representation: Logical, Philosophical, and Computational Foundations, Brooks Cole, Pacific Grove, Calif, USA, 2000..
[6]
G. P. Zarri, “NKRL, a knowledge representation tool for encoding the “Meaning” of complex narrative texts,” Natural Language Engineering—Special Issue on Knowledge Representation for Natural Language Processing in Implemented Systems, vol. 3, pp. 231–253, 1997.
[7]
S. Oda, M. Oda, and M. Yokota, “Conceptual analysis and description of words for color and lightness for grounding them on sensory data,” Transactions of the Japanese Society for Artificial Intelligence, vol. 16, no. 5, pp. 436–444, 2001.
[8]
R. W. Langacker, Concept, Image and Symbol, Mouton de Gruyter, Berlin, Germany, 1991.
[9]
G. A. Miller and P. N. Johnson-Laird, Language and Perception, Harvard University Press, 1976.
[10]
M. Yokota, “Systematic formulation and computation of subjective spatiotemporal knowledge based on mental image directed semantic theory: toward a formal system for natural intelligence,” in Proceedings of the 6th International Workshop on Natural Language Processing and Cognitive Science (NLPCS '09), pp. 133–143, Milan, Italy, May 2009.
[11]
M. Yokota, “Towards awareness computing under control by world knowledge grounded in sensory data,” in Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC '10), pp. 769–775, October 2010.
[12]
B. M. Shariff, M. J. Egenhofer, and D. M. Mark, “Natural-language spatial relations between linear and areal objects: the topology and metric of English-language terms,” International Journal of Geographical Information Science, vol. 12, no. 3, pp. 215–245, 1998.
[13]
P. Roget, Thesaurus of English Words and Phrases, J.M. Dent & Sons Ltd, London, UK, 1975.
[14]
R. Shepard and J. Metzler, “Mental rotation of three-dimensional objects,” Science, vol. 171, no. 3972, pp. 701–703, 1971.
[15]
M. Yokota, “Systematic analysis and synthesis of human subjective knowledge of space and time for intuitive human-robot interaction,” in Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC '11), pp. 208–215, 2011.
[16]
M. Yokota, “Towards artificial communication partners with a multiagent mind model based on mental image directed semantic theory,” in Humanoid Robots, B. Choi, Ed., pp. 333–364, I-Tech Press, 2009.
[17]
J. F. Allen, “Towards a general theory of action and time,” Artificial Intelligence, vol. 23, no. 2, pp. 123–154, 1984.
[18]
D. V. McDermott, “A temporal logic for reasoning about processes and plans,” Cognitive Science, vol. 6, no. 2, pp. 101–155, 1982.
[19]
Y. Shoham, “Time for actions: on the relationship between time, knowledge, and action,” in Proceedings of the International Joint Conference on Artificial Intelligence, pp. 954–959, Detroit, Mich, USA, 1989.
[20]
J. P. Eakins and M. E. Graham, “Content-based Image Retrieval: A report to the JISC Technology Applications Programme,” Institute for Image Data Research, University of Northumbria at Newcastle, 1999.
[21]
M. L. Kherfi, D. Ziou, and A. Bernardi, “Image retrieval from the World Wide Web: issues, techniques, and systems,” ACM Computing Surveys, vol. 36, no. 1, pp. 35–67, 2004.
[22]
M. Yokota, M. Shiraishi, and G. Capi, “Human-robot communication through a mind model based on the mental image directed semantic theory,” in Proceedings of the 10th International Symposium on Artificial Life and Robotics (AROB '05), pp. 695–698, Oita, Japan, 2005.
