JAMES
APPROACH

To help guide the research, JAMES focused on five key objectives:

1. Data collection: To record and analyse the social and task-based behaviour of humans engaged in multimodal joint activities, using a novel Ghost-in-the-Machine data-collection paradigm.

2. Modelling: To design and train a model of social interaction, using annotated data from the human experiments. This model will estimate the social and task-related goals of human partners by processing visual and auditory inputs, and will generate appropriate responses through physical and linguistic actions.

3. Adaptation: To endow the model of social interaction with the ability to learn and adapt to human behaviours, and to handle partial or uncertain information about the state of the world and the mental states of human users.

4. Implementation: To implement the model of social interaction on a physical robot platform, which can operate in an environment with multiple, dynamically changing interaction partners.

5. Evaluation: To evaluate the implemented robot system through physical and simulated interactions with at least two simultaneous human users in a social, task-oriented setting.

The specific demonstration of the JAMES research was a bartender scenario, where the robot played the role of a bartender responding to customers' requests in a dynamic setting, with multiple customers and short interactions. Interactions in the target scenario incorporated a mixture of both task-based and social behaviours, each of which presented certain challenges: a robot existing in the physical world must be able to understand and respond to both the social and the task-based needs of the humans that it encounters, and to successfully distinguish them from each other. Thus, while many human-robot interactions may lead to the same task goal, the quality of those interactions is greatly enhanced by getting the "people skills" right:

Two people, A and B, each individually approach the robot bartender.
Robot (to A): How can I help you?
Person A: A pint of cider, please.
Person C approaches and attracts the attention of the robot by gesturing.
Robot (to C): Just a moment please.
Robot: (Serves A)
Robot (to B): What will you have?
Person B: A pint of bitter.
Robot: (Serves B)
Robot (to C): Thanks for waiting. How can I help you?
Person C: I'd like a glass of red wine.
Robot: (Serves C)

At the research level, the bartender scenario presented challenges for the core components of the JAMES system, by giving rise to situations in which multiple humans are simultaneously present and potentially interacting, both with the robot agent and with each other. This scenario also provided a natural setting where the physical capabilities of a robot platform need only be reasonably sufficient in order to achieve the long-term goals of the project. As a result, this enabled the project to focus its research efforts on correctly developing the core social skills the robot required.