‘Implementation-agnostic’ instantiation schemes for ubiquitous, synchronous multi-user interfaces Dimitrios Kotsalisa George Vellisa Demosthenes Akoumianakisa Jean Vanderdoncktb aDepartment of Informatics Engineering, Technological Education Institution of Crete, Greece bUniversité catholique de Louvain, Louvain School of Management, Belgium Context of the research • Research on advanced, creativity-based User Interface engineering, meaning – – – – – non-trivial case scenarios multi-user interfaces synchronous collaboration distributed users ubiquitous settings • Research protocol – Joint supervision of two researchers of the istlab (Department of Informatics Engineering of TEI Crete) in PhD program at the Université catholique de Louvain, Louvain School of Management, Belgium Plan of the presentation • Case study problem descriptions and overview of solutions (i.e., current version of the implementation) • Brief and non-technical discussion of the engineering challenges and the technicalities of the work – Design-oriented inspiration – Development platforms and tools – On-going work Scenario in MDSD’14 - The virtual setting • Two human collaborators / players – A single shared digital representation • The soccer field – Two types of agents • Human (collaborators) and non-human (soccer players) • Human collaborators virtualized and represented by soccer players – Affordances of soccer players (inspired from semiotic engineering) • Each human collaborator operates with the currently selected soccer player • Each soccer player operates on a shared space inhabited by other players • Each human collaborator operates through different devices (Desktop/Swing and Mobile/Android) • The combined experience (i.e., game co-play) entails operations within different representations • The system in its current operational setting Research question & related works • How can we design and generate non-trivial multiuser interfaces that support synchronous collaboration between distributed users in ubiquitous settings? • Existing approaches – Toolkit-based programming – Model-based UI engineering – Both exhibit limitations which do not allow effective solution to the kind of problem described Proposed approach • Theory-based insights – Interaction devices versus affordances – Semiotic engineering for qualifying virtualities by type of agents and kind of operation • Engineering approach – Development of an abstraction-based model anchoring interaction in to human intentions and capabilities – Specify capabilities in a model-based fashion – Extend UsiXML as needed to support the new specification • Provisions for extensible interaction vocabularies – Build dedicated • design tools • run-time environment components An illustration of the approach • Current implementation of platform-agnostic UI instantiation schemes Platform-agnostic instantiation • A scheme that relies on implementation agnostic (i.e., abstract) specifications of UIs • At run-time and once user and usage context parameters are discovered, the implementation agnostic spec is translated to context-specific interaction vocabularies using dedicated tools Widget gallery Polymorphic classification scheme for the ‘abstract button’ widget Download Download Widget Specification – Basic concept • Introduce new widgets as first-class design objects WSL Libraries Resources Widget Archive Polymorphism & extensible interaction vocabularies Widget Archive An example of polymorphic specification of the ‘abstract SoccerField’ widget • Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages. An example of polymorphic specification of the ‘abstract SoccerField’ widget • Note that polymorphism at the UI-level is a much more demanding notion than polymorphism (i.e., polymorphic method invocation) as implemented in popular Object-Oriented languages. Run-time scenario (MDSD’14 paper) Some technical challenges resolved Input/output techniques and event models Model Breakdowns Different semantics Implementation language/toolkit dependent selected tapped activated View View View Window Window Window Some technical challenges resolved States and state transitions onMouseDown still moving onMouseRelease onSelectCommand onTap still moving onUntap still moving onChangePlayerCommand Sharing schemes for Synchronous collaboration: “Common States” sharing Not abstract enough since there may be no common states or states at all More recent developments (THALES-MusiNet, 2012-2015) Score’ interactive music metaphor: PC, expert users Tablature’ interactive music metaphor: PC, novice users The ‘circular’ interactive music metaphor: Web, novice users Scenarios Multi-user Music Notation Lessons • Distributed Collaborative music learning • User roles – Teacher (Expert) – Student (Novice) JNVT2 • User stereotypes – Sighted – Blind • Platforms – PC: Java/swing – PC: JNVT2 – Web: HTML5 – Prototype • Interactive metaphors – Score – Tablature – Circular JNVT2 Novel features • Advanced collaborative features – Interim-feedback, Group awareness support (Multiuser selection, Multi-user highlighting, Radar view, Filmview), Social awareness HTML5/Prototype • YouTube channel (“MusicNet istLab - Tei Crete”) Polymorphic specification Future work & acknowledgements • Current and on-going work – Support for web2 UIs – Generic support for group awareness – Run-time adaptivity and UI plasticity in distributed and ubiquitous settings – Distributed music learning • Acknowledgements – The work is supported by ARCHIMEDES III, THALES – KUL for PhD dissertations (first two authors) Final remark • All screenshots presented were taken from running prototypes. • For those interested some of them are already available in our channels on YouTube (“MusicNet istLab - Tei Crete”, iSTLab TeiCrete).