Research

Sasha

Smart home assistants function best when user commands are direct–e.g., "turn on the kitchen light"—or when there are hard-coded routines in place to specify the right responses. However, more natural human communication is much less constrained, allowing us to describe goals (e.g., "make it cozy in here") rather than specific actions. Current smart homes cannot understand such commands since they require creative reasoning about devices and settings as they relate to human situations. We approach this problem using large language models (LLMs), exploring their potential to control devices and create automation routines in response to under-specified user commands. We analyze the quality and failure modes of LLM-created action plans in the base case and introduce a key metric–target relevance–for evaluating a model's performance at the task. Finally, we introduce Sasha, a smarter smart home assistant. Sasha responds to commands like "make it cozy" or "help me sleep better" by executing plans to achieve user goals—e.g., setting a mood with available devices, or devising automation routines. We demonstrate Sasha in a real-world implementation and user study, showing its capabilities and limitations when faced with unconstrained user-generated scenarios.

Participants: Evan King (Ph.D. student), Haoxiang (Steven) Yu (Ph.D. student), Sangsu Lee (Ph.D. student), Dr. Christine Julien (MPC director)

Continuous Neighbor Discovery

Identifying “who is around” is key in a plethora of smart scenarios. Our work in this area investigates how to use off-the-shelf devices and technologies (e.g., WiFi direct and Bluetooth Low Energy) to enable lightweight and energy efficient device-to-device continuous discovery. While many solutions exist, they often take a theoretical approach, reasoning about protocol behavior with an abstract model that makes simplifying assumptions about the environment. This approach creates a gap between protocol implementations and the models used during design and analysis. Our work, in contrast is focused on solutions that can be realistically implemented and deployed on real devices and in support of real applications in a way that honestly considers real-world constraints. For instance, our approaches consider the very real effects of packet collisions, which have a real and measurable impact on applications relying on device-to-device neighbor discovery. To our knowledge, this is a first in this domain. Our ultimate goal is to directly empower developers with the ability to determine the optimal protocol configuration for their applications.

Participants: Jie Hua (Ph.D. student), Dr. Christine Julien (MPC director), Amy L. Murphy (Collaborator), Gian Pietro Picco (Collaborator)

Omni Middleware

Device-to-device (D2D) communication technologies are growing in availability and popularity and are commonly used to facilitate applications in Internet of Things (IoT) environments. Such environments are characterized by heterogeneous devices, often employing diverse communication technologies with varying energy consumption, discovery ranges, and transmission rates. These complexities pose a daunting setting for the development of IoT applications that could leverage direct communication with proximal mobile and embedded devices. While current approaches focus either on device discovery in the IoT setting or content transfer assuming established communication channels, none facilitate the intelligent discovery of useful devices and the seamless formation of temporary D2D connections to transfer content directly between devices. Our Omni middleware provides both of these features critical in the development of applications that leverage proximal devices in IoT settings. Using Omni, we demonstrate the feasibility of building applications that use heterogeneous D2D communication channels in an efficient and realistic (in terms of energy and time) manner.

Participants: Eftychia Vakaliou (Ph.D. student), Dr. Christine Julien (MPC director)

Opportunistic Federated Learning

Opportunistic Federated Learning (OFL) is an encounter-based learning paradigm in pervasive environments. We explore how privacy-preserving model training and fine-tuning can be achieved by leveraging neighbor discovery and device-to-device communication in egocentric networks. In OFL, devices opportunistically incorporate intelligence from others in proximity, to obtain the most relevant information as efficiently as possible in terms of communication cost. Our work shows the feasibility of OFL even with extreme skewness in devices' data distributions and diverse personalized objectives.

Participants: Sangsu (Seth) Lee (Ph.D. student), Haoxiang (Steven) Yu (Ph.D. student), Dr. Christine Julien (MPC director)

Project SMART

Project SMART directly and simultaneously seeks to address declining rates of childhood physical activity and to support contextualized computer science and computation thinking (CS/CT) education within the elementary school classroom. The basis of Project SMART is a cooperative educational game that has been adapted to use CS/CT to provide not only learning opportunities but also a focus on the health and educational benefits of physical activity, leveraging organized physical activity opportunities during the school day.

The project relies on community based participatory research to create a researcher-practitioner partnership that includes university researchers in health education and computer science, elementary school administrators and teachers, and K-12 computer science education experts. The combination of physical activity and CS/CT learning in the elementary school classroom is an important novel contribution of this work. There currently exists little emphasis on improving computational thinking in elementary curricula. Computer science pathways that do exist often fail to engage student populations that are traditionally underserved.

Project moveSMART uses a web-based platform to integrate opportunities for physical education with computer science and computational thinking (CS/CT) learning activities. Within the platform, a class cooperatively earns credit for performing physical activity like participating in physical education class or playing outside at recess. This credit takes the form of distances traveled on a virtual path (e.g., along Route 66 across the United States). As the class makes progress, they unlock waypoints that contain CS/CT learning modules as well as additional learning activities directly tied to the required school curriculum. Within the platform, we have developed a series of tutorials designed to introduce elementary students to CS/CT by making connections to physical activity and grade-level curricula in other subjects. Through these tutorials, students create a physical activity monitor using the BBC micro:bit.

Through our work in Project SMART, which also includes professional development sessions for elementary school teachers, we have surfaced multiple challenges that include pressures for all instruction to adhere to required standards, a lack of contextualization of CS/CT content, and unreliable at-home Internet that makes it difficult to reinforce lessons outside of school. By tying CS/CT to students’ own physical activity, we address the dual problems of declining physical activity in children and a lack of contextualization of CS/CT content. To further address identified barriers, we co-designed game elements with classroom teachers to enable cross-curricular connections, including connecting CS/CT to language arts, cultural studies, music, etc.

Participants: Dr. Christine Julien (MPC director), Dr. Darla Castelli (collaborator at The University of Texas at Austin), Dr. Jamie Payton (collaborator at Temple University)

Warble: Middleware for Discovery in the IoT

Advances in sensing and networking along with ubiquitous Internet connectivity have paved the way for today's massive Internet of Thing (IoT) market. Despite the vast potential of connecting to myriad devices across homes, office buildings, and public spaces, there is still a large need to unify the scattered protocols, hubs, and cloud services while personalizing end user experiences. Enabling personalized IoT experiences requires an open infrastructure, backed by expressive and flexible middleware that enables developing applications to address diverse individual needs. Warble is a middleware for personalized IoT applications that encapsulates device and protocol complexities, abstracts communication and IoT devices' impacts in physical spaces into flexible programming abstractions, and enables applications to learn from their prior interactions in the IoT.

Participants: Haoxiang (Steven) Yu (Ph.D. student), Jie Hua (Ph.D. student), Dr. Christine Julien (MPC director), Dr. Gruia-Catalin Roman (collaborator at The University of New Mexico), Colin Milhaupt (collaborator at The University of New Mexico), Hamim Adal (collaborator at The University of New Mexico)

Balance

Balance is an autonomous thermostat control framework that balances a range of user comfort levels (i.e. temperature settings for heating and cooling) with a user-defined monthly budget for heating and air conditioning costs.

Our work is motivated by the large percentage of residential energy usage that is consumed by heating and cooling (HVAC) needs. Many commercial and academic approaches have been developed for autonomous thermostat control to reduce unnecessary HVAC usage. Such systems typically reduce HVAC usage by adjusting the thermostat setting while the house is unoccupied, so as not to disrupt occupants' thermal comfort. We instead consider the balance of the financial burden of HVAC costs with occupants' comfort desires. Balance is an agent-based approach to autonomous thermostat control that is based on the idea that many people are comfortable in a range of temperatures (for example, 68 to 75 degrees Fahrenheit in the summer). Those wishing to meet a certain energy bill each month may wish to adjust the thermostat in order to save money, but mentally converting from the thermostat setting to dollars on an energy bill is challenging, due to the variable impact of weather and household activities on energy used.

Our Balance agent therefore aims to meet the user's monetary HVAC budget by adjusting the thermostat setting within the range of temperatures the user finds comfortable. The agent has two constraints - to meet the user's budget and to stay within the user's comfort range; and one optimization - maximize the user's comfort within that range. To-date, we have implemented a simple agent that accomplishes this task in realistic simulated environments, using real-time HVAC consumption data and indoor and outdoor temperature data to determine thermostat settings at regular intervals throughout each day. Ongoing work on Balance includes: 1) improving the Balance agent's control strategy through reinforcement learning techniques; 2) deploying the Balance agent in the real world, using an Ecobee Smart Thermostat; and 3) investigating the use of feedback, for example if a user provides a budget that is simply not feasible with the user's comfort levels.

Participants: Kate McArdle (Ph.D. student), Dr. Christine Julien (MPC director)

Brace

Brace is a framework that allows developers to explicitly specify both physical and logical assumptions and expected behaviors. Brace then enables run-time checking of these combined physical and logical specifications, provided in the form of assertions, using the physical environment in which a CPS application is running. Brace uses physics models and temporal semantics to guide CPS developers in creating appropriate assertions and to check specified assertions for inconsistencies with the physical world.

Participants: James (Xi) Zheng (Ph.D. student), Dr. Chien-Liang Fok (Post-doctoral researcher), Dr. Christine Julien (MPC director), Dr. Sarfraz Khurshid (collaborator at The University of Texas at Austin), Dr. Miryung Kim (collaborator at The University of Texas at Austin)

Disseminate

Employing opportunistic device-to-device connections among mobile devices has begun to be exploited as a means to alleviate overloaded cellular connections. We have developed a novel publish-subscribe based technique to drive device-to-device data exchange in situations when peers have overlapping interests. Our approach breaks large (media heavy) data into small chunks, and interested mobile peers issue subscriptions and publications over these chunks. An interested device can then receive chunks from the infrastructure or via opportunistic connections to mobile peers. We provide a very lightweight beacon mechanism to exchange interests, and each peer serves chunks of data based on its assessment of the information diversity in the neighborhood of peers. We have implemented our approach in an Android application. The protocol is called Disseminate. Our Android implementation runs on stock Android (assuming support for WiFi-Direct). Using this implementation and through simulation, we have demonstrated that Disseminate improves over state of the art alternatives in terms of overall network overhead and delivery latency.

Participants: Tomasz Kalbarczyk (Ph.D. student), Venkat Srinivasan (Undergraduate student), Dr. Christine Julien (MPC director)

Gander

Gander is a distributed search engine for mobile networked environments characterized by high volumes of short-lived data. We refer to these emerging types of environments as Personalized Networked Spaces (PNetS).

PNetS are comprised of digital devices, both mobile and embedded in the environment (e.g., smart phones, sensors, RFID tags), connected by a dynamic network topology. As in the Internet, large volumes of data motivate the need for expressive search mechanisms that efficiently identify and provide access to information relevant to users' needs. However, the volatile and heterogeneous nature of PNetS' network and data preclude traditional information retrieval techniques.

This project aims to address the novel research and software engineering challenges that arise from the new requirements in this emerging search space.

This project is funded, in part, by the National Science Foundation under grant CNS-0844850 and a Google Research Award. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Participants: Jonas Michel (Ph.D. student), Dr. Christine Julien (MPC director), Dr. Jamie Payton (collaborator at Temple University), Dr. Gruia-Catalin Roman (collaborator at The University of New Mexico)

Grapevine

Grapevine is a framework that enables applications to share context information in a localized region of a pervasive computing network, use that information to dynamically form groups defined by their shared situations, and assess the aggregate context of that group.

Moving beyond more typical egocentric world views, Grapevine allows an application to distribute its own context information while simultaneously leveraging the context information it receives to modify its behavior and aggregate task-relevant group context information that can also distributed within the network. We use novel data structures such as probabilistic Bloomier filters to represent context information efficiently and minimize the network resources required to support Grapevine's use.

Our long term vision is a framework that allows a pervasive computing application developer to delegate all context related functionality to Grapevine and focus solely on the task at hand. Instead of spending time determining what context information is needed, who it should be sent to, and managing the lifecycle of the information it has received, a Grapevine-enabled application can merely indicate the context information it has to offer and the context information it is interested in receiving. Achieving this vision leaves many interesting research challenges such as communicating and responding to interest gradients within the network, determining the frequency with which information should be sent, assessing a quality metric for the context information on hand, and finding ways to provide all this functionality without placing undue burden on the limited resources available to pervasive computing platforms.

This project is funded, in part, by the National Science Foundation under grants CNS-0844850 and CNS-1218232. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

Participants: Samuel (Sungmin) Cho (Assistant professor), Dr. Christine Julien (MPC director)

JIVE - Joint Integration of VLC and Encryption

The vulnerabilities of traditional RF based wireless communications pose a security threat for the transmission of private data among co-located members in physically constrained spaces. Exchanging secure or private information in the clear over direct wireless links is potentially dangerous because eavesdroppers unseen to the communicating parties (e.g., nearby but outside of the room) may overhear the communication. JIVE is a novel encryption key sharing mechanism using Visible Light Communication (VLC). The inherently constrained nature of VLC makes it an ideal medium to communicate secure message encryption keys to individuals in a shared, secure physical space. JIVE can be applied to many scenarios including securing health care information, authenticating smart home IoT devices, and exchanging trade secrets.

Participants: Jayanth Shenoy (Undergraduate student), Aditya Tyagi (Undergraduate student), Meha Halabe (Undergraduate student), Dr. Christine Julien (MPC director)

MadApp

MadApp is an application-level development framework that provides tailored abstractions and support infrastructure for creating dynamic web pages that can incorporate received content from various opportunistic communication channels on-the-fly. Multiple application scenarios exist in which these constructs can seamlessly apply. By constructing expressive exemplars from these domains, we have shown that MadApp can enable opportunistic mobile computing applications to begin to leverage the significant advances in delay-tolerant communication research, opening doors for even more dynamic and adaptive applications.

Participants: Venkat Srinivasan (Undergraduate student), Dr. Christine Julien (MPC director)

MadMAN

Our group does a variety of research related to middleware architectures for delay-tolerant networks. In general, our approaches focus on using adaptation to ensure that nodes in DTNs opportunistically and automatically make use of the available connection opportunities in these highly unpredictable environments. We have performed these adaptations at the "macro" level, adapting the entire network stack from a "traditional" mobile ad hoc network stack to a "delay-tolerant" network stack when conditions dictate such a change. We also explore this a the "micro" level, developing adaptations within the DTN stack that, for example, change the nature and behavior of network coding in response to contextual information.

This project is funded, in part, by the National Science Foundation under grant CNS-0844850 and by the Department of Defense. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsoring agencies.

Pharos

Pharos is a testbed that supports the validation of perception, control, and coordination in pervasive and mobile computing environments. The testbed's physical space, hardware, software, and rich application scenarios support research and discovery at a variety of levels. We support research in sensor management, network coding, wireless protocols, localization, multi-agent coordination, fault-tolerant systems, and software simulation validation.

This project is funded, in part, by the National Science Foundation under grant CNS-0844850 and by the Department of Defense. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsoring agencies.

Participants: Dr. Chien-Liang Fok (Post-doctoral researcher), Dr. Christine Julien (MPC director)