Project Description

LED screensLeehom Wang is an internationally popular, American-born Taiwanese recording artist whose style merges pop, hip-hop, R&B, and traditional Chinese sounds. In 2008-2009, he embarked on his Music Man tour to promote his 13th album, Heart Beat. For this tour, Demiurge Unit, a Hong Kong-based LED design company, was hired for the LED-intensive stage design. The set comprised of seven full-height LED panels for displaying a combination of live video, choreographed animation, and real-time visualization.

Many of the songs has pre-rendered animations created by artists at Demiurge Unit and other collaborators.
Point cloud visualization However, for two sections that featured particular dance performances, they wanted a real-time 3D abstract visualization of those dances. Specifically, they wanted a point-cloud rendering of the dancers that could be panned around, color-shifted, and manipulated with other effects. Due to familiarity with previous work I had done with real-time camera capture, and my ability to create such interactive visualizations on a short timeline, I was brought in to create this system. Although the tour included many smaller stops where the full LED set was not utilized, out interactive system was used in several of the larger shows in Taipei, Shanghai, and Beijing, for audiences of tens of thousands.

My Role

Visualization previewMy primary role was as software developer for this interactive system. This included both the computer vision system for capturing the 3D shape of the dancers in real-time and also visualization of this data. I had less than a month from being contacted until the first show to build this system to meet the expected design, so key was the ability to iterate quickly with the client in the last few days of testing once we were together in Taiwan.

Stereo camera systemIn addition to building the software, I also had to specify all the hardware that needed to be purchased. The important part of this was the camera for fast enough stereo capture and the rest of the camera system which involved infrared emitters and filters. And since the computer would be set up at front-of-house, roughly 200 ft back from the stage where the camera would be, it was important that I had experience picking and using fiber optic cable and fiber optic repeaters that could extend the camera signal that far.

Me operating and calibratingDue to the uncertainty of the lighting conditions in each venue, part of the software was a sophisticated calibration system so that we could adjust camera and algorithm parameters for each city. It was part of my responsibility to fly to each location, set up all the equipment, and properly calibrate the system.

Real-time controlsHowever, during the show, it was Demiurge Unit's creative director who had real-time control of the view, colors, and effects of the visualization. This was due to a physical control system I built, using keyboard and a MIDI control surface, for adjusting parameters. Not only could we adjust these in real-time, but we were also able to test and
Fire video texture effect record preset visual looks during rehearsal that we could easily recall at show time.

Videos

Segment of Shanghai performance