SixthSense

Established: December 26, 2008

RFID based enterprise intelligence

The vision of the SixthSense project in the MNS group at Microsoft Research India is the workplace or home of the future where computing is extended to encompass non-computing entities such as people, objects, and spaces to enable rich user experiences. For instance, we would like users to be able to search the physical world for objects they may have misplaced or use physical events to index their experiences (e.g., “was my laptop with me when I received a phone call from person X?”).

The key technology underlying SixthSense is Radio Frequency Identification, or RFID. The technology comprises inexpensive tags that are attached to objects and readers that are able to read these tags from some distance. RFID is widely used to track the movement of goods through a supply chain. In a typical setting, a reader installed at the entrance of a warehouse can track pallets as they are moved in or out of the warehouse.

In SixthSense, we extend the domain of RFID to settings where there is a rich interaction amongst people and between people and objects. To this end, we consider an enterprise setting of the future where people (or rather their employee badges) and their personal objects such as books, laptops, and mobile phones are tagged with cheap, passive RFID tags, and there is good coverage of RFID readers in the workplace. See the figure below for an illustration of such a setting.

SixthSense provides a platform for RFID-based enterprise intelligence applications. SixthSense combines mobility information obtained from RFID-based sensing with information from enterprise systems such as calendar and presence, to automatically draw inferences about the association and interaction amongst people, objects, and workspaces. For instance, SixthSense is able to automatically distinguish between people and objects, learn the identities of people, and infer the ownership of objects by people. SixthSense also infers when an object has been interacted with by a user, for example, when a a user picks up their mobile phone.

SixthSense is a platform in that its programming model makes the inferences made automatically available to applications via a rich set of APIs. To demonstrate the capabilities of the SixthSense platform, we have prototyped a few applications using these APIs, including a misplaced object alert service, an enhanced calendar service, and rich annotation of video with physical events. We also discuss the issue of safeguarding user privacy in the context of SixthSense.

  • The figure below shows the SixthSense architecture. The key components of the system, including the databases, inference engine, and applications, are run centrally by the enterprise. This provides the (trusted) inference engine access to the complete set of sensed data across all users, objects, and zones, allowing it to make effective inferences. Likewise, the (trusted) application is allowed the flexibility of working with a complete set of inferences (i.e., inferences pertaining to all users and their objects), yet control what processed information is presented to the users to ensure privacy. In contrast, if the inference engine or the application were run by individual users on their own desktop machines, privacy consideration would restrict the set of information made available to these, and hence limit their functionality.

    The main components of SixthSense are:

    RFID Monitor: The RFIDMonitor issues a read command every 500 ms to the RFID reader. The reader reports the EPC and the signal strength (RSSI) of the tags read via each of its antennas. This data then gets pushed into the raw database.

    Other Enterprise Monitors: These monitor the information listed below and push their updates into the the raw database:

    • Calendar Monitor: This resides on each user’s desktop machine, and monitors the time and location of the user’s appointments.
    • Presence Monitor: This monitors the status of each user’s interaction with their desktop. A machine is said to be idle when it receives no user input for 2 minutes. Transitions from idle to active state are detected and reported.
    • Login Monitor: This is similar to the presence monitor except that in general login is a much stronger indication of a user being present than simply a change in their machine’s idle.
    • Cameras: Office buildings are often equipped with cameras for security reasons. The camera feed is stored in a video database for future analysis, if the need arises. We show in applications how we combine the camera system in an enterprise with other sensors to build useful applications.

    Raw database: The RFID monitor and the other enterprise monitors push data into the raw database.

    Inference engine: The inference engine operates on the raw database to draw inferences about people, objects, and workspaces. It makes the following inferences:

    • Person-Object Differentiation: The goal of person-object differentiation is to take an undifferentiated mass of tags and classify each tag as either belonging to a person or to an object. A “person tag” refers to a tag affixed, say, to a person’s employee badge, which is (almost) always carried by him/her.
    • Object Ownership: Having classified tags as people or objects, the goal of this module is to infer the owner (obviously a person) of each object.
    • Zone Identification: We automatically classify RFID zones into individual workspace, shared workspace, reserved shared workspace and common areas.
    • Person Identification: After the nature of tags (person vs. object) has been identified, we then infer the identity of the person corresponding to a person tag.
    • Person-Object Interaction: Finally, we examine events within an RFID zone.Specifically, we infer that an object has been interacted with, e.g., picked up by a person.

    Processed database: The processed database is populated by the inference engine with its inferences, making these available to applications built on top of the SixthSense platform.

    API: SixthSense provides a set of APIs for applications to lookup the inferences stored in the processed database or to receive callbacks when new inferences are made.

  • We have prototyped a few applications using the APIs exposed by the SixthSense platform.

    Misplaced Object Alert

    The misplaced object alert service tracks the movement of users and objects, and notifies a user when it thinks that he/she may have misplaced an object. An object is said to be misplaced when it is in a shared workspace or a public area, and the object’s owner, who was also in the same zone, moves away to a different zone. When such a situation is detected, the system suspects that the object may have been misplaced and alerts the user via email or phone.

    Annotating Video with Physical Events

    We have built a simple application that the annotates the video feed from a camera with RFID events corresponding to inter-zone movement and object interaction. The video could, for instance, be from the security cameras in a building. With the video thus annotated with physical events, a user who is searching for a misplaced objected but suspects that it went missing in the lobby could issue a query like: “find me all segments of the security video recording corresponding to times when I passed through the lobby”. This would then return the desired segments of video, relieving the user of the burden of sifting through hours of irrelevant video recording. A screenshot of the application we have built is shown below.

    Automatic Conference Room Booking

    SixthSense automatically identifies a shared space as a reserved space (e.g., a conference room) using the calendar information of users. While reserving such a room may be the norm, people may sometimes occupy it without reservation, if it is not already occupied and they are looking for space to hold an unplanned, last-minute meeting. Despite the barging in, if the room has in fact not been reserved for the time in question, it would be desirable to reserve it for the group that has barged in, to avoid the possibility of someone else trying to reserve the apparently free but in fact occupied room. We have built an automatic conference room booking application to accomplish precisely this.