Writer / Lung, Chi-Hao (Associate Professor, Depar
National Taiwan Museum of Fine Arts has been devoting to integrating digital technology into artworks and creating more diverse and profound perceptual experiences for its audiences in recent years. With its effort, not only does it turn digital media into great platforms and incubators for displaying and cultivating original art, it also helps to reimagine traditional forms of art and the ideas involved in classic art pieces by creating interactive experiences with digital technology based on them. This time, the museum team reproduced Huang Tu-shui’s Water Buffaloes, a wall-sized
low-relief depicting traditional Taiwanese farms. Engaging with mixed reality (MR) technology, the team crafted a profound experience through the interplay of compression and decompression of artistic perceptions.
Blending together material textures and moving elements with professional sculpting skills, Huang masterfully produced a polymer speaking of the characteristics of both the inner and outer layers, the
materiality, and the spiritual space time. Different distances are clearly perceived, and even an extremely narrow space conveys a sense of depth in his work. In the foreground, the intimate interaction between herd boys and Taiwan water buffaloes are definitely captured, and all kinds of textures are meticulously rendered, while other herd boys, water buffaloes, and Musa basjoo (also know as Japanese Banana) leaves fade into background, making the fields appear to be broad, vast, and misty.
With the use of MR technology, the team presented water buffaloes’ roaming in the field and created a brand-new immersive experience while restructuring the moving parts compressed by traditional
artmaking methods. The details were derived from various elements available for the time the original work was created, including the spatiality, volume, textures, experiences, motions, and the overall vibe formed by all the components involved, all of which were translated into objects attached to the fundamental framework of art and played a critical part for conveying the emotion in the digital version. As the digital mimicry reinterprets the space time crystallised in the original work, it is a decompressed space with the same quality as that of Water Buffaloes. In this sense, the digital version kept the original texture of buffaloes and herding boys to express the s piritual space
time extending from the original work. In the meantime, the team speculated that traditional artists valued skills which helped to represent perceptual experiences, so the team reversely decompressed and interpolated more elements in the key objects, creating a new work with good accordance which does not collide with the perceptual experiences conveyed by the original work.
The medium chosen to present the virtual content is Microsoft HoloLens2, a device that comes with depth sensor and hand-gesture tracker. It is used to present the virtual space time and integrate it with the physical world. To perfectly demonstrate the interaction of the virtual and physical worlds, bridging the gap between spatial experiences and human bodies would be the first and primary issue. The depth sensor and hand-gesture tracker are the two key technologies that open up the possibility of blending the virtual and physical worlds. The following paragraphs would firstly examine how
virtual spaces and physical experiences were connected through the mentioned technologies, which were used as the framework. After that, the three creative aspects brought together by MR technology – recognition, description, and integration – as well as their derivations would also be discussed.
1. Bridging the Gap between Spatial Experiences and Body Movements
On the headset are two signal transmitters and receivers with depth-sensing capabilities, which are also able to track subtle changes of hand gestures and recognise shapes, structures and relative positions of the space users stay in. Images display on the optical combiner of the AR headset through light engines are combining with the ambient light. The virtual elements showing on the combiner are able to overlap with the view of the real-world in real time as users walk around the room, making it possible for water buffaloes to wander in the lobby of the Museum.
2. Recognition: Exploring the Space and Tracking Body Movement
Microsoft HoloLens 2 has depth-sensing capabilities and can recognise the spatial structure and relative positions of a space and other spaces with its build-in accelerometer, gyroscope, and magnetometer. After confirming relative positions of users and other objects around them, hand
tracking is also activated to deliver perfect MR experience. Users can instinctively interact with or create more virtual objects with their hands, and they can also browse the interface and navigate around the virtual tour.
3. Description and Integration: Structuring the Space and Interpolating More Elements in the Virtual Content
This part of the creation is about to re-present the content of the real-world. Traditionally, the nature of art creation lies in the compression of living experiences. The reproduced virtual content, on the other hand, is inspired by the spirit and the artistic quality of the original work, and further interpolated content to deliver a more immersive experience. It was a decompression of the key moving elements.
Narratives of the Virtual Content: The Representation of the Art Making Processes and the Agent of Physical Experience
As MR technology offers abundant real-time feedback for immersive experience, with the advantage of the medium, the team re-presented the creative process of Water Buffaloes and embodied moving moments of the artist. Impersonating the agent of experience and transforming the received sensory
stimuli and impressions, the team granted audiences the opportunity to be “in the artist’s shoes” and take a look at the moving scene the artist experienced.
Artistic Content: Reading and Decompressing the Key Moving Moment
Taiwan water buffaloes, Japanese banana leaves, bamboo hats, and herding boys… All the elements were delicately arranged in Huang’s work. Not only is it an idyll depicting country of the south, but it also showcases the hardworking spirit of Taiwanese people – just like that of Taiwan water buffaloes. The artist managed to showcase the country vibe with the combination of Taiwan water buffaloes and local scenes. (Hsueh Yen-Ling, The Heritage of Taiwanese Art: The Regional Characteristics of Taiwanese Art in the Period of Japanese Rule. Taichung: National Taiwan Museum of Fine Arts,
2004.) The combination of different elements seems to bring to life the temperature, humidity, breeze, and even the sound of rainfall of the time the sculpture was created. Strong connection between body movement and spatial characteristics is vital for creating art experience as it would stimulate a series of sensory experiences and physical movement, while the composition and happenings prompt
people to further explore such experience and its artistic value. Along with the background story delivered by the narrator, the interactive experience made audiences feel like they were in a Japanese banana garden where they could wander around, bend down to walk under trees as the leaves skimming over their heads, and take a closer look at the broad-leaved trees dancing in the wind as if they could have felt the warmth and humidity the artist felt in the air.
There are five different scenes unfolded in Huang’s work with different levels of depth: The closest one is a herding boy riding water buffaloes, and there’s another boy touching the chin of a little
water buffalo behind the first boy, followed by the herd and more boys at the back, composing four of the five layers in the work; finally, Japanese banana trees scatter in the field in the background. Some of these trees are more clearly pictured, overlapping with boys and water buffaloes around them, while other trees fade into the background and convey a vast and misty feel. Each of the five layers provided a coordinate to help the team locate the virtual scenes, and these coordinates formed a continuous area in the mixed reality – overlapping with the lobby of the museum – where Taiwan water buffaloes and herding boys roamed around.
The artist positioned and stacked up different elements to create spatial distance, using low and sunken relief sculpturing skills to demonstrate the different textures and volumes – hard, round, thin, or thick – of the characters and Taiwan water buffaloes, as if they were alive breathing, with their blood running in their vessels. (Huang Tsai-lang, former museum director of the National Taiwan Museum of Fine Arts, 17 June 2013.) All the mentioned elements were reproduced and linked together to represent the delightful movement and steps made by herd boys, the bulky body of water buffaloes and their heavy steps, as well as postures of herd boys on the back of water buffaloes. The portrait of these subjects showcase both their weight and the leisure time they enjoyed. The intimate interaction between herd boys and water buffaloes were reproduced in three-dimensional virtual scenes, allowing audience to move around and look at the scenes from different perspectives, thus
enjoying an art experience unique to themselves.
4. Derivations – The Corresponding Relationships between Actual Space and Virtual Content
The virtual content produced by MR technology responds to the original work, the space, and the way it interacts with audiences. The space in front of the original work, Water Buffaloes, which is exhibited in the museum lobby, was turned into an immersive stage. The virtual Japanese banana garden extending from the work and the creative process of manipulative experience are connected with the real world. All the virtual content derived from the original work is like fruit and blossom, reminding people of the original state of a work as that of a plant while they are appreciating their changes. When experiencing creating process, audience could clearly and succinctly understand the important steps adopted to create a sculpture – from sculpting to moulding – as they could go through every critical process with their own pace and produce a mini relief while immersing in the space. In the experience, all the sculpting procedures were fast forwarded, and subtle changes produced by different techniques were revealed. The simulated objects were instinctively grabbed by audience. The final miniatures were enlarged and blended with the large-sized original work at the back of the stage, reminding audience of the great limit and challenge imposed by the materiality in traditional artmaking processes, as well as the time and effort artists put in when they were young to overcome all kinds of physical challenges to present, preserve, and pass down the moving moment of their lives.
The virtual effect delivered by MR technology has no physical boundary and can transform into a variety of things, but at the end of the day it has to either return to the original, physical state linking with objects, or occasionally showcase some characteristics inheriting from the original work while it is displayed. MR technology integrated with tangible artwork have crystallized the art experience of spiritual space. The imagery and spirit of Taiwan Buffaloes could once again manifest itself instead of freezing the aura, thus motivating and inspiring people to produce artwork with additional dimensions.
Conclusion
New immersive art experiences tap into the space recognition technology and excellent computational skills, like the art of navigation used in the Age of Discovery, inspiring people to further explore the story through revelation and understandings, and hence sailing toward a new continent where other forms of art may be found. With a more comprehensive sensory medium, we emolished, reconstructed, and flipped the way different kinds of artistic perceptions were conveyed. When the heavy chain of materiality can no longer stop us from developing more feelings, the elevation of spiritual space will become the new direction artists of the next generation pursue and exchange with experiences that move them. New perspectives are in need to reflect what the core values are inside the fixed, hard shell of objects. The imagery and ideology freely flowing along and bridging with other elements will be unchained, while a more diversified artistic value will finally be awakened.