- About
- Message from the Chair
- History
- Facilities
- News
- Events
- Info Sci Colloquium
- Technopopulism and the Assault on Indian Democracy
- Generative Agents: Interactive Simulacra of Human Behavior
- AGI is Coming… Is HCI Ready?
- Algorithmic Governance: Auditing Online Systems for Bias and Misinformation
- Studying GenAI as a Cultural Technology: Provocations for Understanding the Cultural Entanglements of AI
- The State of Design Knowledge in Human-AI Interaction
- Understanding “Knowledge”: How Social Epistemology Can Help HCI and AI Researchers to Shape the Future of Generative AI
- How Easy Access to Statistical Likelihoods of Everything Will Change Interaction with Computers
- IS Engaged
- Graduation Info
- Ethics and Politics in Computing Colloquium
- Info Sci Colloquium
- Contact Us
- Courses
- Research
- Computational Social Science
- Critical Data Studies
- Data Science
- Economics and Information
- Education Technology
- Ethics, Law and Policy
- Human-Computer Interaction
- Human-Robot Interaction
- Incentives and Computation
- Infrastructure Studies
- Interface Design and Ubiquitous Computing
- Natural Language Processing
- Network Science
- Social Computing and Computer-supported Cooperative Work
- Technology and Equity
- People
- Career
- Undergraduate
- Info Sci Majors
- BA - Information Science (College of Arts & Sciences)
- BS - Information Science (CALS)
- BS - Information Science, Systems, and Technology
- MPS Early Credit Option
- Independent Research
- CPT Procedures
- Student Associations
- Undergraduate Minor in Info Sci
- Our Students and Alumni
- Graduation Info
- Contact Us
- Info Sci Majors
- Masters
- PHD
- Prospective PhD Students
- Admissions
- Degree Requirements and Curriculum
- Grad Student Orgs
- For Current PhDs
- Diversity and Inclusion
- Our Students and Alumni
- Graduation Info
- Program Contacts and Student Advising
Please join us for the Information Science Colloquium and Computer Science with guest, Morgan Dixon. Morgan is a PhD student in Computer Science & Engineering at the University of Washington working with James Fogarty. His research is centered around designing, building, and studying systems that make user interface software more flexible and extensible. He has also worked on novel interaction techniques for accessibility, dual-display mobile devices, health technologies, crossing-based interfaces, and data analysis tools. Morgan received a B.S. in Computer Science and Mathematics from the University of Maryland in 2008, and an M.S. from the University of Washington in 2010. His work on Prefab received a CHI Best Paper Award, and he was named a Microsoft Research PhD Fellow in 2011.
Talk Title: Unlocking Interaction via Pixel-Based Reverse Engineering
Abstract: User interface software has become a victim of its own success. Graphical event-driven systems are the dominant user interface for computational devices, realized via popular user interface toolkits. However, this success has induced a lock-in effect that demonstrably stifles innovation. For example, human-computer interaction researchers regularly propose new interaction techniques for desktop and mobile devices, enhancements for people with disabilities, better support for localization, and new collaboration facilities. However, these techniques rarely make it outside the laboratory because existing software tools cannot support them. Changing the ecosystem would require a mass adoption of new tools, and most of the applications we use today would need to be rewritten.
I will describe an approach that unlocks existing graphical application without their source code and without their cooperation, making it possible to explore and deploy many promising ideas in the context of existing real-world interfaces. Prefab is a system that reverse-engineers graphical interfaces from raw pixels. I will show how Prefab can be used to augment user interfaces with a variety of techniques, including accessibility enhancements, improved input for touch screens, UI language translation, and end-user customization. Some of these enhancements were proposed in the literature over the last two decades, but prior to Prefab they had been considered nearly impossible to deploy in practice.