Face Mining:
Finding Who and When in Video
| Navigate Video by Character |
|---|
Navigate video by "who" and "when". Click on a character's thumbnails to go to scenes where they appear.
|
|
|
Note: Copyright restrictions limit us to short clips with no sound.
|
|
|
| Browse Characters |
|---|
|
|
|
Travel through space and time with each character. Watch the drama unfold on William Shatner's (Kirk's) face. See if Spock ever smiles. The answer to many more questions are just a click or two away.
|
|
Face Mining Star Trek: Why and How
Our face mining technology enables new video viewing experiences:
- Clickable navigation to scenes of interest
- Search for specific people
- Hyperlinking scenes of interest
- Embedding scenes of interest in other media (e.g blogs)
- Adding commentary synchronized with play points of interests
- Summarizing occurrences of single characters
- Summarizing co-occurrences of characters
Here we illustrate the face mining concept for the TV series Star Trek. Specifically, we applied our state-of-the art algorithms in face detection, face tracking and face recognition to 67 Star Trek episodes over three seasons. This process automatically extracts all visible face tracks, and clusters these into a small number of same-person groupings. Currently, we recognize frontal or near-frontal tracks. In the near future, we will extend our results to non-frontal tracks as well.
Given our face mining output, it took a person less than five minutes to assign names to all the main characters (Kirk, Spock, Mccoy, Uhura, Scotty, Sulu and Chekov) and a couple of minor ones (Janice, Nurse Chapel) across all 67 episodes. The above graph visualizes the relative face time for each of the main characters across all three seasons.
