Practice-Based Research Journal

Week 1

My first experience of VR was an elliptical of awe, shock and horror. I went into the experience believing it to be a journey through the ocean, a peaceful subatlantic exploration where I have a 360 * view of the artificial world around me. Not to say that it wasn’t that, but I was surprised by how much autonomy the Oculus gave me. I felt like I was there. The appearance of the shark caught me off guard.

The Weather Project was an installation created specifically for a particular location. The artist, Eliasson, used a semi-circular screen, a ceiling of mirrors, and artificial mist to give the impression of an indoor sun. To create a giant mirror that doubled the size of the hall, aluminium frames were lined with mirror foil and suspended from the ceiling. The image of a huge indoor sunset was reflected by 200 backlit mono-frequency lights.

Eliasson offers a unique perspective for viewers to immerse themselves in various atmospheres and environments. He achieves this by using techniques such as enlargement and coloured fog to disorient and unfamiliarises the audience’s senses. The installation’s sheer size is noteworthy, as it mirrors the grandeur of the actual sun, allowing the audience to be fully immersed through visual and spatial senses.

This piece was inspiring as it made me reflect on how replication can seem artificial.

It is interesting how images convey an emotion or experience of our world. I want to create something where photography and animation augment reality, immersing the observer.

---

Week 2

Between Body and World	Perception, Illusion, Hallucination
Between Frames	Animation
Between Shots	Cinema and Immersion
Between Screens	Expanded Cinema
Between Objects	Expanded Objects
Between Action and Reaction	Interaction
Between Data Points	The Latent Space of AI

(Mat Wall-Smith, Lecture)

The mind does not reflect truth but rather extracts it from an ongoing process involving the collision and emerging of ideas. 

Crary, J (1988) ‘Techniques of the Observer’ October, vol 45, pp. 3-35 https://www.jstor.org/stable/779041

Explored phenakistoscope further at TraceLoops.

---

Week 3

For this task, my group filmed a 1.30-minute sports sequence of a failing basketballer. This task allowed us to explore coherent sequence with a clear sense of space, time and movement.

We decided to parody the last-second victory trope in movies as our protagonist ultimately fails and is reprimanded. The post-production sound sets the comedic tone.

---

Week 4

Highlights from Sydney Modern Gallery

Highlights from Atmospheric Memory, Powerhouse Museum

---

Case Study – “Borderless” by teamLab

teamLab stands as a pioneering company in the realm of immersive new media, uniting the talents of artists, programmers, engineers, and CG animators to craft an array of inventive art projects (Pen 2021). Through their interdisciplinary approach, they seamlessly meld technology, and human engagement, culminating in immersive exhibitions that resonate with participants on both a physical and sensory level. One prime example of their work is ‘Borderless,’ an all-encompassing exhibit that breathes life into simulated worlds through human interaction (Shi 2020). Noteworthy installations like “The Forest of Flowers and People: Lost, Immersed and Reborn” (2017) and “Universe of Water Particles on a Rock Where People Gather” (2018) transcend spatial confines, inviting guests into a realm where fiction and reality intertwine, forging a deeply personalised mode of engagement. 

“Borderless” wraps visitors within a three-dimensional virtual realm, evoking the authenticity of traditional Japanese scroll art to conjure a sense of genuine presence. This initiative remains at the heart of teamLab’s endeavours in the realm of new media art. teamLab blends digital media and human-computer interfaces to birth distinctive and irreplicable artistic expressions. Their installations strive to facilitate an encounter with the boundless beauty of the real/virtual world and the allure of togetherness (Hua 2021). “The Forest of Flowers and People: Lost, Immersed and Reborn” (2017) defies conventional exhibit viewpoints by infusing the movement and ecological rhythms of the natural world. This creation draws inspiration from the Kunisaki Peninsula, where spring ushers in blooms of cherry trees and rape blossoms (TeamLab 2017). The piece envelops viewers with blossoms that bloom around them, petals gracefully descending as they depart. This dynamic interaction produces an ever-changing artwork, impossible to replicate or revisit (Shi 2020). In this manner, teamLab’s innovative spaces implore ‘viewers’ to engage with more than just sight, revealing the delicate dance of nature and the impact of human presence on its rhythms. Audiences are elicited to employ their full sensory capabilities to bask in the sublime artificial world created by the artists. 

The immersive realm of “Universe of Water Particles on Rocks Where People Gather” (2018) is meticulously designed to kindle a connection between participants and virtual realms. It beckons them to forge relationships with fellow attendees and the artwork itself. Cascading virtual waterfalls grace the screen’s pinnacle, birthing streams that meander across rocky terrain. Water particles alter their course in response to the participant’s trajectory, yielding ripples upon interaction with others (Grassi 2022). This collaborative facet of the piece encourages visitors to ponder their connections within the space. Consequently, the line between art and the observer blurs, and within the immersive experience, the ‘observer’ is stirred by an emotional inquiry that unravels spatial repercussions in which action begets reaction. This generates a profoundly immersive encounter for all participants (Haslem 2020). Beyond embracing multisensory and interactive immersion, each exhibit seamlessly melds art with technology, blurring the border between virtuality and reality, emotion and perception. As participants step into this digital realm, they morph from viewers into subjects, becoming integral components of the waterfall scene – embodied rocks. Whenever individuals linger or interact with the waterfall, they impede the water’s flow, akin to a rock’s obstruction, thereby reshaping the water’s trajectory (Yao 2019). The water’s course continually evolves through human interaction.

The essence of immersive design does not rest in crafting an absolute reality, but in conjuring a simulated realm that stirs audiences’ instinctive multisensory perceptions and resonates with their past encounters. Firmly grounded in the tapestry of traditional Japanese culture, teamLab pays homage to lifelike portrayals of nature, seamlessly interwoven with age-old values, customs, and traditions entrenched in the lineage of Japanese art and expression. “Borderless” encapsulates the distinctive Japanese reverence for interconnectedness. Both “The Forest of Flowers and People: Lost, Immersed and Reborn” (2017) and “Universe of Water Particles on a Rock Where People Gather” (2018) offer immersive havens where viewers become fully engaged through tangible interactions and multisensory experiences, mirroring their established perceptions while concurrently birthing novel encounters and insights.

---

Finding an Idea

I am thinking of exploring an immersive experience that revolves around mental illness. To immerse users, particularly those who have not themselves experienced anxiety/BPD, what it is like. Hopefully, an emotional and disorientating user experience that raise awareness.

Read:

Yip, A 2016, Virtual Reality and the museology of consciousness, Accessed 6 Sep 2023. <https://www.artlink.com.au/articles/4559/virtual-reality-and-the-museology-of-consciousness/>

• Reading made me (re)consider the traditional relationship/s between audience, artist, and artwork.
• Interest in the representation and reception of works
• Artist-driven experiments are a new form of public intimacy
• Immersion unites sensorial components
  • ‘relocating the eye of the camera around the consciousness of the observer.’
• Scopophilic form of spectatorship
• Active stages that aggregate new forms of engagement and interaction within and without [their] walls

---

Nyctophobie

Documentary, Essay

Nyctophobia is an essay-documentary about psychological distress. Jean-François Boisvenue tells us about his childhood fears and his history of mental illness. On the screen, we can see a series of hand-drawn animations projected onto his body that plunge us into psychosis and depersonalisation. The whole was captured by the camera without postproduction effects.

---

Derealisation, from my perspective, is like a sudden hyper-awareness of my current actions and surroundings, accompanied by a profound sense of emptiness and weightlessness in my physical body. I often question, “Am I truly present here, engaged in these activities?” This inquiry tends to trigger a response akin to a mild panic attack, characterised by an accelerated heart rate and escalating anxiety. Reality seems to take on an abstract quality, resembling a video game or the sensation of existing within the Matrix. I begin to feel more like an observer than an active participant, even in my own words and actions. At times, it is as though I’m listening to my own speech as I speak, a sensation I find unsettling, leading me to cease talking altogether. It is as if I’m still inhabiting my body but from a more passive and detached vantage point. Familiar spaces start to warp, with rooms seeming slightly larger or smaller, though pinpointing the exact disparity becomes elusive. Focusing too intently on these distortions exacerbates the experience, compounding the anxiety exponentially.

Through ramblings, I sought to find imagery I could use for a project. A form of idea generation spawned from diary-like entries.

---

---

---

Again, I began thinking about imagery, and the illustrative qualities of verse.

I could not stop thinking about Edgar Allan Poe’s ‘The Raven’.

I began exploring how this piece of gothic literature had been adapted, curious about its possible properties as a transmedia text.

---

Understand the Poem:

• Begin by analysing its themes, characters, and the dark, melancholic atmosphere it evokes. 
• This understanding will guide my creative decisions.

Choosing the Medium:

• Decide on the medium of your immersive project.
• Options include virtual reality (VR), augmented reality (AR), 360-degree video, interactive websites, etc.

Conceptualise the Experience:

• Brainstorm how you want to translate ‘The Raven’ into an immersive format.

Develop Storyline:

• Create a narrative or concept that captures the essence of ‘The Raven.’
• Decide how you will adapt the poem’ plot and characters to fit your chosen medium.

Script and Dialogue:

• Write a script for your immersive experience, incorporating dialogue and narration that stay true to Poe’s poetic language and style.

Audio Design:

• Sound effects, ambient sounds and music should enhance the emotional impact of the experience.
• Consider incorporating the poem’s verses into the narration.

Visual Design:

• Plan the visual aspects of your project, including set design, lighting and effects.
• Ensure the visuals evoke the eerie and gothic atmosphere of the poem.

Interactive Elements:

• What interactive elements can I incorporate to engage users while advancing the story?

Production:

• Begin production by shooting video, creating 3D assets, recording sound, or developing interactive components.

Post-Production:

• Edit and refine your content, ensuring that it aligns with your vision.

Testing and Debugging:

• Test the immersive experience extensively to identify and resolve any issues, such as technical glitches or user experience problems. 

---

Week 6

This week was immensely helpful in conceptualising my project. The projectors would make a fine asset to my project.

---

Read:

Kurtzman, E 2019, Immersed in Horror: A Study of the Historical and Contemporary Influences of Poe’s Shadows, MA Thesis, Virginia Tech

https://vtechworks.lib.vt.edu/handle/10919/91183

• Combined elements of traditional theatre, original art, and innovative technology to present an immersive experience of Edgar Allan Poe’s work.
• https://vimeo.com/306222169
• Garden used directional speakers and large scrims to immerse visitors in Shakespeare’s sonnets and scenes.
• Horror is a genre marked by the invocation of fear and repulsion.
• As far back as 1420, inventors and engineers were seeking ways to project images into the air and create ghostly effects.
• The 20th century has marked the rise of cinematic arts.
• The artwork, which was hand-drawn, was then rendered into animation.
• Evocative images
• The images are mostly static and black and white, but there are also subtle movements within the pictures as well as spots of thematic colour.
• Noise surrounds the audience on all sides.
• ‘Poe’s Shadows provided its visitors with a rare chance to participate in a journey through darkness actively.’
• Poe’s Shadows projected black-and-white illustrations onto a cyclorama screen.

San Juan, E 1967, The Form of Experience in the Poems of Edgar Allan Poe, The Georgia Review, Vol. 21, No. 1, pp. 65-80

https://www.jstor.org/stable/41396331?casa_token=jnFQcCoMDS8AAAAA%3AdCUdT33FB-JcsrYhtd6R7xKzzWiI9QPqrgMVOZOh-FSzMBhNdyU42wWpM3Hb2iQ335tKgpylhSNFeE0J8CeioZLiPkDzfGDNajviYd47CDWlupqiP7g2&seq=2

• Poetry is “the Rhythmical Creation of Beauty”
• The Raven’s eyes reveal the interior outlook of the soul.
• ‘Nevermore’ is concerned with concealment and enwombing.
• The void needs an antithesis to define itself e.g. good to evil.
• Vague psychic states
• Self-hypnotic
• Unvarying pentameter
• A dream within a dream.
• Absurdity of languishing over unfilled hopes.
• ‘Spiritual ectasy and prophetic rapture stem from a cognition of material limitations’
• The imagery pursues a circular movement.

---

Choosing the Medium

I want to do something with either 360* video or Augmented Reality through projectors and speakers like the week 6 tutorial.

Read:

Rall, H 2020, Redefining Visual Storytelling for Adaption of Classic Literature in Immersive Environments: Holderlin’s Echo VR, Culture and Computing, International Conference on Human-Computer Interaction, pp. 229-244.

https://link.springer.com/chapter/10.1007/978-3-030-50267-6_18

---

Conceptualise the Experience

https://www.youtube.com/watch?v=t_Nu8BkjjJ0&ab_channel=MeinVideoStudioEN

Read:

Ferdwig, E 2020, Effect and influence of ambisonic audio in viewing 360 video, Journal of virtual worlds research, National Science Foundation, Vol. 13, No. 2-3, pp. 1-14.

https://par.nsf.gov/biblio/10274524

Siddig, A, Ragano, A, Jahromi, H & Hines, A 2019, Fusion confusion: exploring ambisonic spatial localisation for audio-visual immersion using the McGurk effect. Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems pp. 28-33.

https://doi.org/10.1145/3304113.3326112

---

Develop Storyline

---

Key: [ ] = audio output, BOLD = higher frequency; amplified

[music + ambiance begin]

Once upon a midnight dreary, while I pondered, weak and weary,

Over many a quaint and curious volume of forgotten lore—

While I nodded, nearly napping, suddenly there came a tapping,

[tapping 3:3]

As of someone gently rapping, rapping at my chamber door.

“’Tis some visitor,” I muttered, “tapping at my chamber door—

 Only this and nothing more.”

Ah, distinctly, I remember it was in the bleak December;

[fire crackling]

And each separate dying ember wrought its ghost upon the floor.

Eagerly I wished the morrow;—vainly I had sought to borrow

From my books surcease of sorrow—sorrow for the lost Lenore—

[harp – key change]

For the rare and radiant maiden whom the angels name Lenore—

            Nameless here for evermore.

[curtains rustling, wind]

And the silken, sad, uncertain rustling of each purple curtain

Thrilled me—filled me with fantastic terrors never felt before;

[heart beating]

{giggles}

So that now, to still the beating of my heart, I stood repeating

“’Tis some visitor entreating entrance at my chamber door—

Some late visitor entreating entrance at my chamber door;—

            This it is and nothing more.”

Presently my soul grew stronger; hesitating then no longer,

“Sir,” said I, “or Madam, truly your forgiveness I implore;

But the fact is I was napping, and so gently you came rapping,

And so faintly you came tapping, tapping at my chamber door,

That I scarce was sure I heard you”—here I opened wide the door;—

[silence, BEAT.]

            Darkness there and nothing more.

[wind, soft]

Deep into that darkness, peering, long I stood there wondering, fearing,

[wind, rising]

Doubting, dreaming dreams no mortal ever dared to dream before;

[Wind, halting]

But the silence was unbroken, and the stillness gave no token,

And the only word there spoken was the whispered word, “Lenore?”

[BEAT.]

This I whispered, and an echo murmured back the word, “Lenore!”—

            Merely this and nothing more.

[fire crackling, clothing rustling]

Back into the chamber turning, all my soul within me burning,

Soon again,

[tapping, louder 3:3]

 I heard a tapping somewhat louder than before.

“Surely,” said I, “surely that is something at my window lattice;

      Let me see, then, what thereat is, and this mystery explore—

[wind]

Let my heart be still a moment and this mystery explore;—

            ’Tis the wind and nothing more!”

[Shutter opening, harshly]

Open here I flung the shutter, when,

[Bird fluttering]

 with many a flirt and flutter, In there stepped a stately Raven of the saintly days of yore;

Not the least obeisance made he; not a minute stopped or stayed he;

But, with mien of lord or lady, perched above my chamber door—

Perched upon a bust of Pallas just above my chamber door—

            Perched, and sat, and nothing more.

Then this ebony bird beguiling my sad fancy into smiling,

By the grave and stern decorum of the countenance it wore,

“Though thy crest be shorn and shaven, thou,” I said, “art sure no craven,

Ghastly grim and ancient Raven wandering from the Nightly shore—

Tell me what thy lordly name is on the Night’s Plutonian shore!”

            Quoth the Raven “Nevermore.”

[Music rise, breathing heavy]

Much I marvelled this ungainly fowl to hear discourse so plainly,

Though its answer little meaning—little relevancy bore;

For we cannot help agreeing that no living human being

Ever yet was blessed with seeing bird above his chamber door—

Bird or beast upon the sculptured bust above his chamber door,

            With such name as “Nevermore.”

But the Raven, sitting lonely on the placid bust, spoke only

That one word, as if his soul in that one word he did outpour.

Nothing farther then he uttered—not a feather then he fluttered—

Till I scarcely more than muttered “Other friends have flown before—

On the morrow he will leave me, as my Hopes have flown before.”

            Then the bird said “Nevermore.”

Startled at the stillness broken by reply so aptly spoken,

“Doubtless,” said I, “what it utters is its only stock and store

Caught from some unhappy master whom unmerciful Disaster

Followed fast and followed faster till his songs one burden bore—

Till the dirges of his Hope that melancholy burden bore

            Of ‘Never—nevermore’.”

But the Raven still beguiling all my fancy into smiling,

[Seat legs sliding across floor]

Straight I wheeled a cushioned seat in front of bird, and bust and door;

[Velvet sinking]

Then, upon the velvet sinking, I betook myself to linking

Fancy unto fancy, thinking what this ominous bird of yore—

What this grim, ungainly, ghastly, gaunt, and ominous bird of yore

            Meant in croaking “Nevermore.”

[Fire crackling, louder]

This I sat engaged in guessing, but no syllable expressing

To the fowl whose fiery eyes now burned into my bosom’s core;

This and more I sat divining, with my head at ease reclining

[Velvet sinking, lamp buzzing]

On the cushion’s velvet lining that the lamp-light gloated o’er,

But whose velvet-violet lining with the lamp-light gloating o’er,

            She shall press, ah, nevermore!

[Heavy breathing]

Then, methought, the air grew denser, perfumed from an unseen censer

Swung by Seraphim whose foot-falls tinkled on the tufted floor.

“Wretch,” I cried, “thy God hath lent thee—by these angels he hath sent thee

Respite—respite and nepenthe from thy memories of Lenore;

Quaff, oh quaff this kind nepenthe and forget this lost Lenore!”

            Quoth the Raven “Nevermore.”

[Thunder strike]

“Prophet!” said I, “thing of evil!—prophet still, if bird or devil!—

Whether Tempter sent, or whether tempest tossed thee here ashore,

Desolate yet all undaunted, on this desert land enchanted—

On this home by Horror haunted—tell me truly, I implore—

Is there—is there balm in Gilead?—tell me—tell me, I implore!”

            Quoth the Raven “Nevermore.”

[wind, stronger]

“Prophet!” said I, “thing of evil!—prophet still, if bird or devil!

By that Heaven that bends above us—by that God we both adore—

Tell this soul with sorrow laden if, within the distant Aidenn,

It shall clasp a sainted maiden whom the angels name Lenore—

Clasp a rare and radiant maiden whom the angels name Lenore.”

            Quoth the Raven “Nevermore.”

“Be that word our sign of parting, bird or fiend!” I shrieked, upstarting—

“Get thee back into the tempest and the Night’s Plutonian shore!

Leave no black plume as a token of that lie thy soul hath spoken!

Leave my loneliness unbroken!—quit the bust above my door!

Take thy beak from out my heart, and take thy form from off my door!”

            Quoth the Raven “Nevermore.”

And the Raven, never flitting, still is sitting, still is sitting

On the pallid bust of Pallas just above my chamber door;

And his eyes have all the seeming of a demon’s that is dreaming,

[lamp buzzing]

And the lamp-light o’er him streaming throws his shadow on the floor;

And my soul from out that shadow that lies floating on the floor

            Shall be lifted—nevermore!

---

Week 7

The lecture explained audio in a way I understood 🙂 (I was still lost up until then)

---

Audio Design

Recorded narration (Script).

Began audio mixing.

---

Critical Reflection

My immersive media project reimagines Edgar Allan Poe’s iconic poem, ‘The Raven’, through the use of projectors and speakers. It exemplifies a captivating intersection between literature and technology. 

The project’s most notable strength lies in its ability to leverage technology as a portal to the poem’s atmospheric depth. The combined utilisation of projectors and speakers masterfully crafts an all-encompassing sensorial experience. This immersive environment evokes the eerie ambience of gothic literature, drawing the participants into Poe’s world. The dark visual projections intertwined with haunting auditory elements serve as a dynamic medium for immersion.

This immersive adaption capitalises on technology’s prowess to visually represent the intricate themes and symbolism woven throughout ‘The Raven.’ The projectors breathe life into the motifs (the raven, the chamber door, and the narrator’s psychological state), rendering these literary elements tangible and more comprehensible. The result is a multi-layered exploration of Poe’s work, deepening the participants’ appreciation and understanding of its underlying themes. 

The project could integrate interactivity to further explore the intersection of literature and technology. Allowing participants to engage with the narrative or environment actively could elevate the immersive experience. Such interactivity would promote a deeper exploration of the poem’s themes and symbolism, turning the audience into active participants rather than passive observers. 

The immersive adaption of Edgar Allan Poe’s ‘The Raven’ represents a captivating confluence of literature and technology. Its ability to evoke the poem’s atmosphere, visually represent its themes and leverage technology for immersive storytelling is all there. However, the project could be more immersive if I explored interactivity. This project can push the boundaries of how technology and literature intersect by embracing interactivity, creating a dynamic and enriching literary experience.

---

Creating assets/ images.