Evaluating Experiences on Connected Devices


Modern TV watching behaviour often involves using a mobile device while viewing a programme often leading to a more interactive experience across multiple screens.

Due to the complexities in designing connected multi-device experiences, such as the multifaceted relationship between individuals and their content, the uncertainty of user intent or motivation while using devices (shared vs. personal), the diversity of environments in which users might be based and so on, it’s difficult to focus scope into one research study.

In this project, I focused on how tolerant users are to different levels of “simulated” time delays using a representative set of passive, non-interactive experiences presented on two supposedly synchronised screens – a TV and a Tablet. The Tablet would either be in sync with the TV or lag behind it.

In collaboration with…

Other groups within my organisation (BBC R&D) and IRT. Funded by the EU project 2-IMMERSE.

Design Research Questions

The two main research questions are

  • How perceptible are users to simulated delays in synchronised companion screen experiences?
  • How much delay is tolerable while viewing different types of synchronised companion screen experiences?


Multi-modal experiences

The three genres chosen for the study were Factual, Sports and Drama.

Even with a constrained focus on genre, companion screen experiences come in different forms. In order to limit the number of conditions tested in an exploratory study and present a variety of experience types, the design has been whittled down to three (3) experiences:

  • Video-to-Alt-Video: A Sports programme on a TV with a video stream filmed from an alternative camera angle on a Tablet.
  • Video-to-Audio Description: A Drama programme on a TV with an audio description stream on a Tablet.
  • Video-to-Slideshow: A Factual programme on a TV with a complimentary slideshow on a Tablet.


A companion screen experience will always require the user to manage splitting their attention between two (or more) streams of content. Therefore, the design of connected experiences has to consider the rhythm of the programme content, the pace of information on the TV, the social issues related to the programme and an understanding of how the service is used in a home environment amongst other things. In collaboration with industry partners, I led the user research on how to customise and personalise the experience of viewing programme content on connected TVs in tandem, with additional (companion) content on a personal device (such as a mobile phone or tablet).

Design Approach

Taking into account lessons learnt in previous studies, informal conversations with programme makers and audience research explorations, six genres were initially chosen as being suited to companion screen experiences: ‘Factual (Documentaries)’, ‘News’, ‘Sports’, ‘Drama’, ‘Gameshows’ and ‘Childrens’.

I further whittled down the choices through looking for simple companion experiences which a broadcaster might offer as a regular but useful service with minimal additional effort involved in the production pipeline while still maintaining near production quality authenticity and reusing as much of the existing content produced, broadcasted or published by a broadcaster.

Video to Alt Video

Colleagues in the Sports Library were able to identify a rugby match (England versus France six nations 2015) in which the players managed to score 11 tries which gave me enough options to create test materials. These were created using the broadcasted coverage of the match, on the TV, coupled with a video stream of an alternative camera coverage of the tries on the Tablet.

Companion sample: Video-to-Alt Video (Sports)

Video to Audio Description

This was the most straightforward experience to craft. The existing audio description of the programme was simply stripped off the broadcasted material and delivered on the companion screen. Most of the effort went into selecting a drama within the iPlayer archives – the “Crows” episode of ‘Wolf Hall’ – which had numerous candlelit scenes and longer than usual non-verbal interactions between characters.

Companion sample: Video to Audio Description (Drama)

Video to Slideshow

The programme chosen for this use case was the “Winter” episode of ‘Alaska: Earth’s Frozen Kingdom’ due to its slow pace and predictable easy-to-digest rhythm. The companion content for this was created in collaboration with programme producer, a UX designer and an illustrator. Factoring in the complexity of programme scene, the producer created a storyboard which guided the illustrator and the UX designer in crafting the companion slideshow. The resulting slides differed in visual complexity depending on the ebb and flow of the programme

Companion Storyboard: Video-to-Slideshow (Factual Programme)

There were eight clips of content chosen for each of the three genres of programmes (Factual, Sport, Drama) selected for this study amounting to 24 clips in total. Each of the 24 clips was associated with companion content.

Levels of Delays

Initially, a wide range of delays were chosen to cover the three very different types of experiences chosen for this study. The 0ms delay was chosen to represent the control condition in the study. Colleagues were asked to take part in pilots to evaluate if the levels of delays chosen and the experiences crafted were suitable. Eight levels of delays per genre of programmes were chosen for the experiment.

Evaluative Approach

Since there were 3 types of experiences presented against 8 levels of synchronisation delays, a full repeated measure study would have required each participant to watch 24 varying experiences. In order to minimise participant fatigue, a modified Latin square with repeated measure experiment design was chosen. Each participant was administered a pre-assigned randomised set of 12 clips out of a set of 24 possible clips, of varying delays, of which 4 were Factual, 4 were Sports and 4 were Drama.

Participants were briefed at the begining of the study and asked to sit and ‘watch’ twelve experiences. They were asked to fill in a post-condition survey after watching each of the 12 experiences. After the participant finished answering the twelfth post- condition questionnaire, they were asked to complete a post-study questionnaire which focused on their thoughts on the three different types of experiences. The participant was then interviewed in a semi-structured manner.


The study revealed different levels of delays which participants were prepared to tolerate for different types of experiences and quite some insights into the preferences participants fed back about the experiences.

Details of the project is available in my TVX 2018 paper and related details about the system can be read in my TVX 2016 paper (which won an honourable mention for the best paper award). I supervised two Masters students (Christoph Ziegler from LMU and Wei Liang Kenny Chua from UCL) in very similar ideas. Their projects are detailed in a TVX 2017 paper (which won the best paper award) and a TVX 2017 poster paper. I was also involved in demonstrating the concept of companion screens and the underlying standards (DVB-CSS / HbbTV 2.0) work that makes it possible at IBC where it was featured on ‘what caught my eye’.

Presenting / Hosting as ACM TVX Technical Program Chair
Celebrating a successful conference
Best Paper award ACM TVX 2017