Computational frameworks for interactive multimedia art installations in public spaces using real time digital control and visualization

Draft/Patch/Weave

The following document describes an artistic endeavour combining modular synthesis with floor loom weaving by González-Toledo et al. (2023). The project’s central concept is the weaving draft, which uses Max/MSP to generate and comprehend here-and-now weaving instructions. This approach opens the path for performance-based linkages between modular synthesis and weaving, two fields typically unrelated. Driven by data and sensory interactions between digital and analog worlds, the system displays fresh approaches of creative expression via weavings, patches, and compositions.

Scheme for Max

Scheme for Max (S4M) integrates Scheme with Max/MSP to increase the platform’s capacity for precise event scheduling and live scripting by Cox (2021). This is an open-source application. By directly integrating Scheme functions with the Max scheduler, the principal approach provides exact control over musical tempo. Apart from that, users of the Max environment can script and assess code, providing further opportunities for generative design, real-time composition, and music computing.

Avendish and declarative media interfaces (ADMI) in C++

This article exposes problems with current abstraction techniques and offers a declarative approach for media processing using modern C++ reflection by Celerier (2022). The proposed approach uses a non-intrusive subset of the object model instead of class-based inheritance to overcome framework-specific restrictions when specifying media processors. The Avendish package is a nice illustration of this approach since it creates user and application programming interfaces (APIs) and automatically binds to interfaces like VST, Max, and Open Sound Control (OSC) during compilation.

Low-cost plant bio-signal interface

It presents a simple and cheap interface for transforming electrical signals from plants into digital data usable in creative media applications constructed with Max/MSP by Duncan (2021). Technology that records biosignals in their natural condition, free of anthropomorphic interpretations, is available for less than $5. These signals are used in real-time installations to start or regulate motion, music, or film. Through multimedia art and low-barrier, scientifically responsible methods of bio-interface, our technologies promote creative, polite interactions across species.

Binaural rendering toolbox

Designed for psychoacoustic research and 3D sound exploration, the Binaural rendering toolbox (BRT) is a SONICOM-developed modular software suite. The Avendish package provides C++ libraries for simulating listener-source interactions and environmental acoustics and links to well-known frameworks like Max/MSP and VST by Weinberg & Bockrath (2024). A breakthrough is its capacity to enable reproducible audio experiments using dynamic SOFA standards for geographical data. The BRT is an all-inclusive virtual lab for binaural audio research under OSC control.

The suggested MAX/MSP-TD system integrates real-time sensor input, auditory analysis, and visual output. It addresses media synchronization and latency in public interactive systems. Modular, reusable pieces enable rapid prototyping and environmental adaptation.

In our public testing show in Table 1, we effectively turned user movement and sound into live projections.

Consequently, viewers and digital artists benefit from responsiveness and new creative expression outlets.

Xia et al. (2024) proposed the S-O-R framework to explore the relationship between digital art exhibitions and psychological well-being among the Chinese Generation. Engagement in online digital exhibits may improve Generation Z users’ mental health, according to one study. Based on the Stimulus-Organism-Response (S-O-R) framework and restorative environments theory, this study explores Generation Z participants’ psychological reactions to online digital art exhibits, especially website aesthetics. We also examine how these reactions affect location attachment and loyalty. Using structural equation modeling, an online digital art show was launched on the popular Chinese Generation Z app ZEPETO. The show was followed by an online survey with 332 verified replies. The results show that: (1) the four design elements of website aesthetics (coherence, novelty, interactivity, immersion) significantly influence Generation Z users’ perceived restoration, with immersion being the most influential; (2) perceived restoration and place attachment are crucial predictors of loyalty behavior; and (3) perceived restoration positively affects Generation Z users’ place attachment to online digital art exhibitions. This research shows that online digital art exhibits might help Generation Z recover emotionally and reduce stress. Additionally, digital technology displays may transcend human creativity and imagination, providing a new and promising avenue for emotional healing, design study, and practice.

Interactive multimedia art installations in public settings are limited by latency, scalability, and flexibility using existing frameworks like S4M and BRT. First, many systems delay user inputs, such as sound or motion, and visual outputs, making latency a major concern. This hinders real-time involvement, which is crucial for immersive experiences in dynamic public spaces. Second, these frameworks’ inability to properly handle new input data or devices limits scalability. Thus, elaborate, large-scale deployments diminish performance. The lack of freedom for artists to design interactive aspects limits creative potential in these systems. They are generally limited to predetermined templates or visual effects, making it difficult to change artistic or environmental demands. Following this, the MAX/MSP-TD framework can be introduced as a solution, emphasizing how it overcomes these shortcomings. For instance, it offers low-latency processing (under 150 ms), a modular design that scales effectively without performance loss, and a high degree of flexibility, allowing for dynamic, user-defined visual effects. By clearly contrasting the limitations of existing systems with the capabilities of the MAX/MSP-TD framework, the novelty and advantages of the proposed system will be evident, providing a more critical and focused synthesis of the literature.

Framework architecture and design principles

The proposed system’s basic structure is a scalable, highly flexible, and easily understandable architecture for interactive multimedia projects. Designed with two well-known visual programming tools, Max/MSP and Touch Designer, the framework is easily expanded, upgraded, or combined using its modular components.

This method introduces a visual component that makes the process more approachable for designers and artists, facilitating simple prototyping and a creative iterative process, as shown in Fig. 1. Unlike monolithic systems, which are rigid and scalable, the emphasis here is on reusability and plug-in compatibility. The architectural design supports synchronized control across many modalities by helping to lower latency and smoothly integrate real-time data sources. This section presents the fundamental concepts that enable the framework to be technically robust, flexible, and artistically friendly.

(1) $$|s_i(C^{+}{(CN)}^m|=|{(CN)}^l{\alpha }_i\left(C^{+}\right)+m_i|\left(CN\right).$$Equation (1) shows how the modular components of the system $s_i$ are interdependent. Although every node is in charge of a different chore $C^{+}$ (like data collecting from sensors, audio processing, or graphic creation), it clarifies the relationships among them $(CN{)}^m$. The system is seen as a network $CN$ of related functions whereby the output of one module $m$ serves as an input for another. The modular $m_i$ architecture guarantees a great degree of versatility and simple reconfiguration $(CN{)}^l$.

(2) $$-div(s\left(D\right)s\left(\left|t\right|\left|\mathrm{d}\right|r{s}^{-2}{\mathrm{\nabla }}_v\right)=a\left(D,t,\mathrm{\nabla }d\right).$$Regardless of whether from sensors $s$ or audio sources, Eq. (2) depicts the real-time data $D$ flow across the system $s$, therefore exposing the transformation $t$ and processing of input data $d$ at every stage. Equation (2) was validated using microphone latency tests at a 48 kHz sampling rate. Consuming raw sensor $rs$ input, the equation shows the stages (such as normalization, filtering, and analysis) used to convert it into acceptable audiovisual $a$ outputs by consumption, processing, and finally exporting the results.

(3) $$\underset{d\to \mathrm{\infty }}{lim}\Vert \left(H_{iR}(v_n\right)-\left(H_{ia}(u\right))|{\mathrm{\partial }}_i\left(u_I\right){|}^{ia}{\mathrm{\partial }}_i\left(u_{ai}\right){\Vert }_{L{\displaystyle \frac{cs}{cs-1}}}=0.$$Equation (3) helps us understand how the system’s auditory and visual elements cooperate $\left(H_{iR}(v_n\right)$. Due to real-time data $d$ processing, it demonstrates how perfectly harmonic. $H_{iR}$ the created video $v$, audio $ia$, and lighting are $\left(u_I\right){|}^{ia}$. Visual material, for example, may change in response to user $u$ gestures or auditory impulses $I$ or vice versa; consequently, it is essential that both components $cs$ be handled simultaneously by ${|}_{L{\displaystyle \frac{cs}{cs-1}}}$.

(4) $$\underset{L\to \mathrm{\infty }}{\mathrm{l}\mathrm{i}\mathrm{m}}snd⟨I_F\left(u_e\right)-L_c\left(I\right),u_e-L⟩\le 0.$$Equation (4) describes an interactive feedback loop $L$ in which the user’s activities, such as movement or sound $snd,$ impact the system, influencing $I$ the user’s next actions. Fundamental $F$ to interactive exhibitions $u_e$ is the capacity $L_c$ of the system to adapt to continuous input, since it generates a constantly changing experience.

These technological limitations restrict the responsiveness and efficiency of interactive systems and the creative freedom of designers and artists. Reacting to these challenges, the study presented here offers MAX/MSP-TD, a Node-based Real-Time Multimedia Architecture developed using Max/MSP, as shown in Algorithm 1. This approach provides a customizable, real-time system that combines generative graphics, audio analysis, and sensor data utilizing various visual programming platforms. The framework’s modular construction and adaptability help to enable a broad spectrum of interactive public art situations, therefore fostering general public innovation and engagement. This design paradigm enables the development of responsive, intelligent systems, as each node or module is responsible for a specific purpose, such as data collection, auditory processing, or visual production.

Integration of real-time data and multimedia outputs

The framework’s backbone is its ability to receive, interpret, and react to data inputs in real-time from a range of microphones and sensors; this lets public interaction be instantly converted into audiovisual feedback. Max/MSP handles the audio processing, signal routing, and data manipulation chores;

This integration enables installations to behave like living, adaptive creatures by rapidly responding to changes in their surroundings or human action, as illustrated in Fig. 2. Here, we observe how the system generates innovative outputs by absorbing data from the outside world; we also see how modular patching allows for more expression flexibility without compromising efficiency. The result is an immersive, multisensory experience, making it challenging to identify who is observing and actively participating.

(5) $$\underset{n\to \mathrm{\infty }}{\mathrm{I}\mathrm{n}\mathrm{p}}enV⟨(E_S-M)\left(P_n\right),A_n-S⟩\le 0.$$Equation (5) shows how inputs $\mathrm{I}\mathrm{n}\mathrm{p},$ such as motion, sound, or environmental $enV$ elements $E$ from real-time sensors $(E_S$ (such as motion, sound, or environmental factors) may be translated into related multimedia $M$ outputs, such as projected $P_n$ visuals or audible $A_n$ reactions. The system constantly seeks to convert sensor data into interactive multimedia. This mapping ensures that the audiovisual outputs $enV$ would react in real-time to any changes in user input or outside stimuli.

(6) $$S(P,f{)}_k={\displaystyle \frac{1}{3}\left[S_a{\mathrm{\Delta }}_n{X}_k+{\mathrm{\Delta }}_z{(Pf)}_a\right].}$$Equation (6) shows simultaneous $S$ transformation, processing $P$, and filtering $f$ of raw sensor $S$ data; it is then used to create multimedia outputs. Sensor data must be filtered or normalized to be easily accessible $a$. This is particularly true for noisy $n$ data, including light levels, movement vectors, or audio frequencies. Following preprocessing, the updated data is used to influence multimedia’s visual effects and sound frequency.

(7) $$E_{ocur}\left(H\right)=\mathrm{\partial }+\sum _{T=1}^C{A}_i\left(Oi+1\right).$$Electronic $E$ outputs, including visual projections by Eq. (7) and sound, must occur $ocur$ simultaneously and in harmony $H$ with real-time data, and this equation describes the synchronization process that accomplishes just that. For example, when the user moves the cursor $C$ to trigger $T$ an audio effect, the visual output should complement the audio $A$ in terms of rhythm, timing, and overall impact $Oi$. The synchronization of the auditory and visual components guarantees that they respond to the same data stream and work together as one cohesive system.

TD generates dynamic graphic content that depends on real-time stimuli. The systems coordinate audio and video output using standards like OSC. The system can convert human inputs, ambient noises, and motion into audio textures and immersive presentations.

Application in urban installation and system performance

Applying the proposed framework in a real-time urban interactive art show showed great success. The system recorded passers-by in the real world using ambient microphones and motion sensors.

It also assesses the system’s responsiveness, latency, user involvement, and capacity for surrounding adaptation, illustrated in Fig. 3. Maintaining multimodal synchronization and guaranteeing low-latency performance was significantly simpler with the MAX/MSP-TD architecture than with conventional systems. The modular architecture of the arrangement allows for simple modification to fit different locations or audience behavior. Participant comments verified a higher degree of participation and interest, and the system’s capacity to transform inactive urban environments into dynamic platforms for artistic expression. This case study is the ideal illustration of the technical and creative viability of the framework.

(8) $$R^{m+1}\le \left(1+S\tau \right)s^r+\tau i^{a}Gd=0,1,2,3,\dots ..$$Equation (8) shows the system’s response $R$ to a user’s input, that is, a movement $m$ or sound $S$ with a suitable display or sound $s$. Keeping this delay low would help keep things looking realistic $r$ and interactive installations in real-time, appealing $a$. If the user engages with the installation for 30 s, this metric logs the time as 30 s. This metric assesses how long the user stays engaged, indicating whether the installation is compelling or if latency makes it frustrating. During real-time performance tests, the system logs the time difference between the user’s input and the visual output. If the system shows a latency of 0.06 s (which is slightly higher than the desired 0.05), this could indicate a need for optimization in processing speed to maintain interactivity. The formula considers the time needed to get data $Gd$, analyze it, and produce the intended outcome.

(9) $$C-1\left(0\right){\alpha }_{mXn}=C-1\left(0\right)\omega gm{P}_0=C-1\left(0\right)\cup {\cap }_0.$$Equation (9) clarifies how computing $C$ tasks are distributed across the several modules $m$ or nodes $n$ to maintain the system running at maximum performance $mP$. A balanced load in large-scale urban installations with significant processing demand provides flawless operation free of lag or system malfunctions. Distribution of processing power among visual rendering, audio processing, and sensor data management takes the front stage here.

(10) $${\left(Sd\right)}_t+u_f\left(c,e\right):={\displaystyle \frac{\int Z^{3}Me\left(S,d\right)ce}{\int Z^{3}Me\left(S,d\right)ce}.}$$In the urban installation in Eq. (10), the conversion of spatial data $Sd,$ user $u$ position or crowd movement $c$ are utilized to map to environmental zones $e$ are expressed in Eq. (10). Based on the user’s behavior and sensor data, many multimedia effects $Me$ are activated by various zones $Z.$ It is crucial for large-scale performance that demands for location-based user interaction over enormous physical areas.

Corresponding light patterns and audio responses are generated by processing these inputs in RT, and it is projected into the public sphere. Here, the technical configuration, interaction flow, and installation settings are clearly described. For creating an interactive multimedia art in public areas, the MAX/MSP-TD was utilized, and it will create a Node-based Real-Time Multimedia Architecture. The drawbacks in the current methods can also be resolved using this modular model. The seamless RT integration of sensor data, auditory inputs, and generated images are all facilitated by its modularity. Dynamically converting public movement and sound into an immersive audio-visual experiences for audience can be done with this application in RT urban installation.

Analysis of system responsiveness

Responsiveness is measured by end-to-end delay, which refers to the time between a user’s input (e.g., a gesture or sound) and the system’s output (visual or auditory response). In this installation, the end-to-end delay was consistently under 80 milliseconds, ensuring near-instantaneous feedback that contributes to a seamless interactive experience.

Below, we examine the system’s response speed to user input, focusing on how quickly it offers visible and audible feedback and its low latency, as represented in Fig. 4. Smooth public interaction in continually changing environments depends on a more responsive MAX/MSP-TD architecture. The performance of the installation is evaluated using measures like system responsiveness, stability under varying interaction loads, and qualitative user input. The MAX/MSP-TD system greatly reduces latency and improves system responsiveness and scalability. In many urban backgrounds, high personalization and easy creativity are promoted by this application. The RT public multimedia systems are well-founded in both art and technology, and it was indicated by the studies. The sample size was chosen based on prior studies in the field, where a minimum of 30 participants per group is typically sufficient for achieving a power of 0.8 in most behavioral studies. To provide more robust insight into the data, confidence intervals (CIs) were calculated for each key metric, such as latency and interaction duration. For example, the mean latency for the MAX/MSP-TD framework was found to be 120 ms, with a 95% confidence interval of [115, 125 ms]. This indicates that, based on our sample, we are 95% confident that the true mean latency for the system lies within this range. Similarly, the interaction duration for the same system had a mean of 240 s, with a 95% CI of [230, 250 s], providing a clear indication of the precision of the measurements.

Empirical validation using latency data or user interaction logs enhances credibility, showing real-world system performance against theoretical models. Additionally, a comparative analysis with existing tools, such as Pure Data or openFrameworks, demonstrates how the proposed system optimizes real-time processing or improves user responsiveness, which positions it effectively within the landscape of interactive multimedia frameworks.

Analysis of processing efficiency

The effectiveness of real-time sensors and multimedia data handling is key to this study, as illustrated in Fig. 5. Its node-based modular design improves processing capacity and lowers bottlenecks, enabling flawless multimedia transitions and real-time control.

Analysis of user interaction feedback

We now turn to discuss user engagement and enjoyment. We asked for comments to assess the intuition, responsiveness, and immersion shown in Fig. 6. The results reveal that the interactive projections and adaptive feedback of the system extend the length of the interaction and increase user enjoyment. Furthermore, the modular architecture of the framework is tested for handling different spatial and sensory configurations. This study of MAX/MSP-TD offers insightful information that supports the framework’s capacity to address the regular problems of synchronizing, inflexibility, and latency in multimedia public art displays. User engagement is quantified through interaction duration logs, which track the total time users spend interacting with the installation. For instance, the average interaction time for users across different sessions was found to be approximately 7 min, with some users engaging for up to 15 min. This indicates a high level of interest and suggests that the installation captures and sustains attention effectively. This study was approved by the Institutional Review Board (IRB)/Ethics Committee of The Catholic University of Korea, Performing Arts and Culture Department (IRB Approval Number: 1040395-202503-24). The study complies with all ethical standards for research involving human participants, including informed consent, confidentiality of participant data, and the right to withdraw at any time without penalty.

The study acknowledges several limitations related to measurement precision, potential bias, and sampling constraints that may influence the interpretation of the findings. First, the sensors used to capture user activities and ambient circumstances may have limits that impair measurement accuracy. While calibration methods reduced mistakes, tiny sensor data imperfections might still add unpredictability, particularly in fast-paced or dynamic scenarios. The sample may not completely reflect the public since the participants were from a particular demographic with previous interactive system experience. Selection bias may affect the generalizability despite attempts to broaden the participant pool. Standardized methods were followed to prevent experimenter bias, although minor signals or expectations during data collection may have introduced it. The relatively modest sample size of 30 individuals per condition may not reflect the entire range of user behaviors in bigger or more varied groups, but it is sufficient for statistical power analysis.

Biological implications of this work include how environmental conditions affect population adaptive qualities. The study examines how selection stresses, such as environment or food availability, cause phenotypic alterations that improve survival and reproduction. The increase in beak size in birds in response to tougher seeds is an example of directional selection, when a feature confers a fitness benefit in a certain environment. By studying the genetics of this feature, the research ties phenotypic changes to evolutionary processes and illuminates adaptation’s genetic underpinnings. A power analysis establishes the minimal sample size needed to detect significant differences in attributes. For this research, 150 participants per group are selected because they have enough statistical power (0.8) to identify moderate effect sizes at 0.05.

IRB permission is required for the study, guaranteeing that it follows all established ethical guidelines for research involving human subjects. To ensure that participants are fully aware of the recordings, their purpose, and the study’s intended use of their data, researchers use either digital or article forms to gather their informed consent. Individuals can decline participation without penalty through a clear opt-out mechanism that is integrated into the consent process.