ARTS 2961: Designing Musical Games

Arts Department, Rensselaer Polytechnic Institute

Monday/Thursday, 12:00-1:50pm, VAST Lab, SAGE

Instructor: Rob Hamilton, West Hall 114

e: hamilr4 [at] rpi [dot] edu


Office Hours: ONLINE





One of the most exciting areas of music technology development is happening in the realm of gaming and interactive virtual space. Music and Sound Design play crucial roles in the design of gaming environments, narratives and flow. And as designers create ever more innovative game experiences featuring rich graphics, fast multiplayer networking and next-generation controllers, new techniques for creating immersive music and sound for games to complement and showcase these advances are not only possible but necessary.

This Studio class will explore cutting edge techniques for building interactive sound and music systems for games and 2D/3D rendered environments. To better understand the link between virtual space and sound, students will learn the basics of designing sound and composing music for interactive game spaces by designing and implementing rich musical games within the Unreal gaming engine. Coursework will require the ability and desire to code game logic and design game environments. Techniques for integrating sound and music within games including game-centric middleware tools like FMOD and WWISE, interactive sound synthesis and computer networking using Open Sound Control will all be explored.

Working in teams or on their own, students will design their own music-rich game experience, compose music, design sound material, and implement their own playable musical game experiences.


Students will explore the artistic role of music and sound in gaming by building their own interactive sound and music-rich games and 2D/3D rendered environments. Within the context of their own creative game projects, students will learn the basics of designing sound and composing music for interactive game spaces. Using workflow programming languages and software tools, students will program basic gaming interactions, link them to interactive audio software, and create a musical gaming experience.


    Students who successfully complete this course will demonstrate...
  1. an understanding and appreciation of game design and musical creation through an awareness of the many disciplines underlying the field including: software design and programming, interaction design, listening skills, musical theory, musical acoustics, digital audio theory, and digital signal processing.
  2. basic technical facility in the areas of game development, audio recording, editing, sound synthesis, software development and post production.
  3. creativity and resourcefulness through the creation of musical game environments and composition of your own sonic projects


Evaluation is based on the following:


You will be required to present some of your assignments to the class, to show your work within the software environment you used to create it, and to engage the class in discussion of your work. When you are not presenting your own work, you need to be attentive to whoever is presenting, and to engage them in discussion of their work. Failure to participate in class will lower your grade.


You must attend class to succeed in this course.

  1. Since much of the class is focused on listening to and discussing work in class, attendance is mandatory.
  2. ** More then two unexcused absences will affect your grade, detracting 1/2 grade each additional 2 unexcused absences. **
  3. Absences can only be excused by a letter from a medical doctor or from the Dean of Students' office.
  4. Late arrivals are very disruptive - continued late arrival will affect your grade.
  5. It should go without saying but no use of mobile devices or personal computers during class time (except for as required by the coursework itself) is acceptable. Continued violations will be treated as an unexcused absence.


Collaboration between students in this course is strongly encouraged. Likewise, students are encouraged—indeed, to some extent required—to exchange ideas, opinions and information . You are also encouraged to help each other in the lab and with performance, production, and presentation of composition projects.

Plagiarism of any kind is in direct violation of University policy on Academic Dishonesty as defined in the Rensselaer Handbook, and penalties for plagiarism can be severe. In this class you will be expected to attribute due credit to the originator of any ideas, words, sounds, or music which you incorporate substantially into your own work. This applies particularly to citation of sources for sonic "samples" included in your compositions.

Submission of any assignment that is in violation of this policy may result in a grade of F for the assignment in question. Violation of this policy will be reported, as defined in the Rensselaer Handbook


Students requiring assistance are encouraged to contact Disability Services: to discuss any special accommodations or needs for this course.


Office Hours for Fall, 2021 will be ONLINE ONLY


Designing Sound, by Andy Farnell

Musimathics (Vol. I), by Gareth Loy

Musimathics (Vol, II), by Gareth Loy

Theory and Techniques of Electronic Music, by Miller Puckette

ARTS 2961 Git Repo:


The proposed course topics and schedule will be as follows (take note of project due dates!). Based on class progress and interests, this schedule is subject to change. Special topics, guest lectures, supplemental reading, listening and additional assignments to be announced.

Week 1:
Monday, 8/30
aka: Kanye-day (Ye-day? Ye-de?)

Introduction: Designing Musical Games :: Gaming Musical Design
- The role of sound and music in games; filmic influences; spectra of interactivity

Welcome to DMG Questionaire: Goo-drive link

Thursday, 9/2
aka: Trent-day

Sound/Audio Overview:
- Sound Design, Music, Digital Audio, MIDI, Sample rate,

Tool Overview:
- Audacity:
- Unity:
- Unreal engine:
- Pure Data (PD):
- ChucK:
- WWise:

What is Game Sound?

- Classic Video Game Sounds Explained by Experts (1972-1998) Pt. 1 | Wired
- "The resolution of sound: Understanding retro game audio beyond the ‘8-bit’ horizon"

Audacity: normalization, editing

Assignment #1: In Class/At Home: Sound Design/Editing Challenge (
Your challlenge, should you choose to accept it*, is to create a set of sounds for your next AAA game project using only your voice, and Audacity.

    Using whatever recording hardware is currently at your disposal and Audacity, create 5 variations of 5 different sound events listed below (Extra Credit: do as many of the following sounds as possible):
      Shooting sound
      Rolling ball
      Large object impact
      Small object impact
      Short looping musical sequence (pitched "pad"" sounds)
      Low drum (kick)
      High-pitched drum (tom)
      Sharp attack drum (snare)
      Motor idling
      Motor revving
      Space-ship flying
      HUD interface sounds (button presses, toggles, sliders)
      Player Death
      Player Spawn
      Game Victory
      Game Failure
Week 2:
Monday, 8/30
aka: CherYe-day

Introduction to Unity: tutorials
Unity new input system:
Videos: Audio Listeners and Sources
Unity Third Person Example: here
Audio Listener
Audio Source


CherYe's "Do You Believe I'm Heartless"

Thursday, 9/9

Video : Digital Foley: "Mortal Kombat Foley: The Sound of Violence"
Video: "The Hunger Games" & "Frozen" Foley Artists Turn the Sound of Junk into Miracles

Simple Sound Design - Audacity + Unity
- Normalization, Fade in/out, basic editing

Audacity FFMpeg Library:

Arrays of AudioClips - Randomization of Parameters
Footsteps 101

Limbo Walkthrough:
Week 3:
Monday, 9/13
aka: Black Francis day

Audio Mixer
More Audio Tutorials
Techniques for Fading Audio
Manual Link: Audio

Unity Audio pt. II: AudioMixers and AudioGroups, exposing parameters, filters/FX, scripting audio playback

3D Sound, Volume, Audio Filters

Assignment #2: Sample-based game audio scene (Due Thursday, 9/23, 11:59:59 PM AoE [anywhere on earth] )

Using your favorite samples from our in-class recording challenge, populate our class game scene with your sounds. You can modify and add to the scene as needed/desired to make the game do what you'd like it to.

The success of your scene will revolve around whether or not your scene feels coherent, i.e. all the sounds exist in the same space. Coherence can be helped by creative use of reverberation in game, by careful editing of sounds in Audacity, or simply through artistic choices you make in your role as sound-designer.
In addition to the project itself, create a detailed "call-sheet" list of all the sounds you will use/are using in the project, including variations. List file-names, extensions, durations, and any notes that would be helpful or useful to your programming team*. Please include a short paragraph or description of the overall-zeitgeist of your scene, and why we should care.
Please include examples of the following Unity audio processes in your scene:

    * Simple Audio Source playback (multiple)
    * Mixer Groups
    * Basic C# scripting of audio playback events
    * Varying pitch, amplitude, etc.
    * 3D sound/distance-based attenuation
    * Sounds triggered through collision
    * A looping stereo background "musical" track
    * A multi-channel "stem"-based playback example
    * EXTRA-CREDIT: include MIDI file control and/or Microphone input

Please upload your entire project to your Box folder.

* the programming team is a lie...

Thursday, 9/16
Class Music Stems: Gouge Away (rob), Ye
Samples: N64 Samples (Reddit), Drum samples (ChucK)

Reverb Zones
AudioMixer bussing (continued): Sends and Receives, Ducking (Sidechaining), Snapshot transitions, Mixer Parameterization
Advanced Mixing

Audio as Data: Microphone input - thread...
Microphone input

Audio Optimization (In-Engine):

Asset Bundles: intro, Building Asset Bundles
Profiler (Audio)
Week 4:
Monday, 9/20
Assignment #2 hack-day - bring your questions.

Thursday, 9/23
Data systems for Music and Gaming

Audio data: Fourier Analysis
Audio Analyzer
FFT/Pitch Detection: AudioListener.GetSpectrumData

Beyond MIDI: Chpt. 2
What Is Midi?

Easy-to access Note Events:

MidiJack: MIDI Input to Unity
C Sharp Synth for Unity
Unity Midi

Sonic Visualizer: Visualisation, analysis and annotation of music audio recordings
Unity Text File read/write: StreamReader, TextAsset

Week 5:
Monday, 9/27
Musical Game Design Session #1: Gamelan

Thursday, 9/30
Introduction to Unreal Engine 4

Blueprint Workflow Scripting Language
IDE basics, Sound Wave Assets: importing, properties
Audio Listener - Blueprint for moving from Camera to third-person character
Week 6:
Monday, 10/4
Unreal Audio (continued)

Git repo (in Discord)

Footsteps via animation Blueprint (anim notify):

Sound Cues, Sound Cue Editor:

Box/Capsule/Sphere Triggers:
Week 7:
Monday, 10/11
Thursday, 10/14
Unreal Submixes/Submix Effects continued...

Real-time Synthesis in Unreal - GDC 2017 Presentation
Modular Synth basics:
UNREAL Midi Input - default midi input classes
githargyro 6-channel MIDI input test project (in github)
Week 8:
Monday, 10/18
Soundsnap: free edu login codes in each of your box folders
Open Sound Control:
Introduction to Pure Data:
Pure Data and Spore
Kent Jolly, Aaron McLeran @ GDC 2008:

Assignment #3 Proposal Document (Due Saturday, 10/23, 11:59:59 PM AoE [anywhere on earth] )
* Submit a written project proposal for your Assignment #3 Unreal Engine audio project (see below). This proposal should describe the scene you'll be creating, the types of sound assets and sound generation techniques you plan on using, and should show me that you've given thought to the overall direction and focus of your upcoming assignent.
* There is no "minimum" or "maximum" length to your design document submission; if you can succinctly explain your project in one sentence, all the better. However it is your responsibility to clearly explain your proposed project and the technologies you will leverage to create it to me.

Assignment #3: Unreal audio scene (Due Monday, 11/1, 11:59:59 PM AoE [anywhere on earth] )

This project represents your chance to showcase your understanding of the basic underlying principles of game sound and music within the Unreal Engine. Populate a simple Unreal scene using sound files downloaded from Soundsnap or some other online repository - or sound assets of your own creation (even better!). You can create an Unreal scene from scratch or use one of the many available Unreal project templates (displayed when you create a new Unreal project).

As always, the success of your scene will revolve around whether or not your scene feels coherent, i.e. all the sounds exist in the same space. Coherence can be helped by creative use of reverberation in game, by careful editing of sounds in Audacity, or simply through artistic choices you make in your role as sound-designer. In addition to the project itself, create a detailed "call-sheet" list of all the sounds you will use/are using in the project, including variations. List file-names, extensions, durations, and any notes that would be helpful or useful to your programming team*. Please include a short paragraph or description of the overall-zeitgeist of your scene, and why we should care.

Please include examples of the following Unreal audio processes in your scene:

    * Footsteps (for 1st and 3rd person scenes), or movement sounds (i.e. vehicle sounds)
    * Simple Sound Assets
    * Sound Assets that continuously play regardless of player distance
    * Randomized Sound Cues
    * Sound Assets/Cues with proximity volume attenuation
    * Sound Assets/Cues triggered by Box or Sphere trigger volumes
    * Sound Assets or Cues triggered by a keypress
    * Sound Cues with one or more parameters driven by in-game action or events
    * Audio Submixes
    * Audio Submix Effects
    * EXTRA-CREDIT: MIDI input, use of the Unreal Modular or Granular Synths, Open Sound Control output or input

Please upload your entire project to your Box folder in a zipped up sub-folder titled "Assignment #3".

Thursday, 10/21
Unreal Audio Continued...
Week 9:
Monday, 10/25
Introduction to Pure Data Sequences: [metro], arrays, tabread, notes, [line~]
Farnell Pure Data Examples (book):  
Thursday, 10/28
Bidirectional OSC communication in Unreal: OSC listeners
How to setup OSC listeners in Unreal:
Pure Data sequences continued...
Journey Playthrough - conclusion.
Week 10:
Monday, 11/1
Pure Data Continued...
Thursday, 11/4
Pure Data Continued...
Week 11:
Monday, 11/8
Unity OSC output with OSCSharp
Bi-directional OSC PD Patch: simple_pd_receiver.pd -
OSCsharp C# classes, integration with Unity -
PD + Unity: Roll-A-Ball OSC demo project -
PD read data from files (abstraction) -
PD sequencer Example -
Thursday, 11/11
Pure Data wrap-up: encapsulating PD
Ultra-mega-ok-pd-tutorial-patch: -
Installing PD externals:
More Pure Data Tutorial Patches: -
Martin Brinkmann patches:
RJDJ Abstractions:

Tuesday 11/23, 11:59:59 pm AoE - Final Project Proposals Due: Submit a detailed written 1-page description of your proposed final project for this class. Your project may encompass any or all of the sound and music compositional/music/sonic techniques and technologies we have investigated during the course of this class including fixed media sound design and composition with Audacity or Pro Tools, workflow interactive programming systems with Pure Data, Unity, Unreal and/or Open Sound Control. Your proposal should address the integration (technical) and the mapping (artistic) of your data and control systems from game engine to audio layer, and vice versa. Your proposal should be clear, concise and well written (typed) and should include both creative ideas and descriptions alongside technical tools and concepts.  
Week 12:
Monday, 11/15
Introduction to WWise: Cube demo, Limbo
Roll-a-ball No Audio:
GDC Talk: Martin Stig Andersen: INSIDE: A Game that Listens:
Rollaball with WWise Project:
Thursday, 11/18
WWise continued: Events with multiple sounds, random containers
Unity/WWise Integration basics
Week 13:
Monday, 11/22
Coretet: Demo Video
Coretet: SIGGRAPH Live performance
Trois Machins de la Grâce Aimante: VR String Quartet
Elegy (Ready, Set, Rapture): for VR Doublebass
Carillon: for VR performers and laptop orchestra
ECHO::Canyon: sonification of avatar physiologies
ECHO::Improvisations: live performers and game system
Tele-Harmonium: for piano and virtual performer
Jord og Himmel: for q3osc and laptop orchestra
Maps and Legends: for q3apd
Thursday, 11/25
NO CLASS (Thanksgiving Holiday)  
Week 14:
Monday, 11/29

Link Dump:

Installing externals: link
Sound-only games: Soundvoyager, Papa Sangre
Listening games: dotstream, Enemy-Zero

ALT-CTL: Alternate controllers for musical games (midi keyboards, wiimote w/osculator
Fragment String:, src code:

Social Composition: Musical Data Systems for Expressive Mobile Music
Unity Teapot: Alvin Lucier's "Nothing is Real" for game engine and Leap Motion
Leap Motion, hid)
FractOSC walkthrough:
Reactable: demo, mobile
Auraglyph: demo

More Musical games: Sentris, FractOSC, Proteus, Panoramical

Rotating Brains Beating Heart - Avatar Orchestra Metaverse: video
Canned Bits Mechanics by Juan Pablo Caceres, Rob Hamilton: video
Appolonian Gasket by Ignacio Pecino: video, paper
Singularity by Ignacio Pecino: video, demo (Firefox, Safari)
Kilgore by Marko Ciciliani: video
BEEP: Game Sound Documentary (festival cut)
Thursday, 12/2
Week 15:
Monday, 12/6


Thursday, 12/9