SWARM (WORKING TITLE)
SUMMARY
Swarm (WORKING TITLE) is aimed to be an interactive experience where the user can use hand tracking to interact and watch fish flock and swim around hands.
This is a solo project done through Unreal Engine's Niagara System connected with MediaPipe Solutions (Google AI Edge) through Python.
TOOLS
Unreal Engine (Niagara Particle Systems, Materials, Scratch Pad), HLSL, Python
YEAR
2026
Concept
Idea
The goal of this project was to create an interactive experience that combined hand tracking with visuals. Watching fish can be very soothing and though this would work the best as an immersive VR experience, webcams are more accessible so I chose this direction for a proof of concept. So for this project the plan is thatI would explore computer vision (hand tracking), Unreal’s Niagara Particle Systems and flock/swarm algorithms.
I did the initial coloured sketch in Procreate.
Unreal Engine Niagara Particles
Fish Shape
I created these fish through flat ribbon particles. The age of the particle would create the shape of the fish and I would use materials to create the illusion of detail and depth for each ribbon.
Using the texture coordinates of the ribbon and some maths, I refined the fin shape, coloured the fish, and gave them eyes. I used dynamic parameters to add some slight variation in colour and fin shape.
Simulation / Leader Particles
The Niagara System is set up so that there is a Leader particle emitter and the fish and ripple emitters are spawned based off of the location of each leader particle.
The leader particle system has simulation stages that check over each particle to deal with any collisions with each other. This is optimised with spatial grid partitioning through a NeighborGrid 3D system, which divides the Niagara System into localised grid cells. Each particle then only checks particles stored in neighboring grid cells for any simulations instead of comparing each particle to all other particles. The swarming of the particles is created through the use of the boid simulation. The majority of this is done through creating custom scratch pad modules to run HLSL code.
There were a lot of small things that I wanted to adjust to my system that were too fiddly to achieve with Niagara’s default modules. So I used some scratch pad and HLSL to add a sine offset to mimic a swimming animation and some slight colour variation to the flock..
Python / MediaPipe
Hand Tracking
MediaPipe is an open-source framework developed by Google. It has many solutions for real-time computer vision. I used one to track hand landmarks for each frame from a webcam stream.
Joints of the hands and the palm's location from the webcam screen is outputted and processed through Python.
Python to Unreal Engine
To get all this hand tracking data over to Unreal Engine, I had to build a nested OSC (Open Sound Control) bundle to handle all the information. This data type is quick and perfect for sending over something real-time and Unreal supports it.
On the Unreal Engine side, I started a OSCServer to receive the data bundles to then parse it to use it within my project.
