Search
Search results for: multimodal
-
Multimodal Communication
Journals -
Multimodal Attention Stimulator
PublicationMultimodal attention stimulator was proposed and tested for improving auditory and visual attention, including pupils with developmental dyslexia. Results of the conducted experiments shown that the designed stimulator can be used in order to improve comprehension during reading tasks. The changes in the visual attention, observed in reading test results, translate into the overall reading performance.
-
Multimodal Technologies and Interaction
Journals -
Journal on Multimodal User Interfaces
Journals -
Multimodal English corpus for automatic speech recognition
PublicationA multimodal corpus developed for research of speech recognition based on audio-visual data is presented. Besides usual video and sound excerpts, the prepared database contains also thermovision images and depth maps. All streams were recorded simultaneously, therefore the corpus enables to examine the importance of the information provided by different modalities. Based on the recordings, it is also possible to develop a speech...
-
Virtual touchpad - video-based multimodal interface
PublicationA new computer interface named Virtual-Touchpad (VTP) is presented. The Virtual-Touchpad provides a multimodal interface which enables controlling computer applications by hand gestures captured with a typical webcam. The video stream is processed in the software layer of the interface. Hitherto existing video-based interfaces analyzing frames of hand gestures are presented. Then, the hardware configuration and software features...
-
FEEDB: A multimodal database of facial expressions and emotions
PublicationIn this paper a first version of a multimodal FEEDB database of facial expressions and emotions is presented. The database contains labeled RGB-D recordings of people expressing a specific set of expressions that have been recorded using Microsoft Kinect sensor. Such a database can be used for classifier training and testing in face recognition as well as in recognition of facial expressions and human emotions. Also initial experiences...
-
Wireless multimodal localization sensor for industrial applications
PublicationThis paper presents the concept and design of a wireless multimodal localization sensor for hybrid localization systems combining vision-based, radio-based and inertial techniques in order to alleviate problems in harsh and complex industrial environments. It supports two radio technologies, 868 MHz UHF RFID and 2.4 GHz WSN, for positioning purposes and communications. The sensor includes LED light transmitters for vision-based...
-
New Applications of Multimodal Human-Computer Interfaces
PublicationMultimodal computer interfaces and examples of their applications to education software and for the disabled people are presented. The proposed interfaces include the interactive electronic whiteboard based on video image analysis, application for controlling computers with gestures and the audio interface for speech stretching for hearing impaired and stuttering people. Application of the eye-gaze tracking system to awareness...
-
Multimodal Surveillance Based Personal Protection System
PublicationA novel, multimodal approach for automatic detection of abduction of a protected individual, employing dedicated personal protection device and a city monitoring system is proposed and overviewed. The solution is based on combining four modalities (signals coming from: Bluetooth, fixed and PTZ cameras, thermal camera, acoustic sensors). The Bluetooth signal is used continuously to monitor the protected person presence, and in case...
-
Selection of Features for Multimodal Vocalic Segments Classification
PublicationEnglish speech recognition experiments are presented employing both: audio signal and Facial Motion Capture (FMC) recordings. The principal aim of the study was to evaluate the influence of feature vector dimension reduction for the accuracy of vocalic segments classification employing neural networks. Several parameter reduction strategies were adopted, namely: Extremely Randomized Trees, Principal Component Analysis and Recursive...
-
Multimodal Platform for Continuous Monitoring of the Elderly and Disabled
Publication,
-
Multimodal Audio-Visual Recognition of Traffic Events
PublicationPrzedstawiono demonstrator systemu wykrywania niebezpiecznych zdarzeń w ruchu drogowym oparty na jednoczesnej analizie danych wizyjnych i akustycznych. System jest częścią systemu automatycznego nadzoru bezpieczeństwa. Wykorzystuje on kamery i mikrofony jako źródła danych. Przedstawiono wykorzystane algorytmy - algorytmy rozpoznawania zdarzeń dźwiękowych oraz analizy obrazu. Zaprezentowano wyniki działania algorytmów na przykładzie...
-
Multimodal platform for continuous monitoring of elderly and disabled
PublicationArtykuł opisuje założenia do realizacji wielomodalnej platformy monitoringu osób starszych i chorych
-
Multimodal learning application with interactive animated character. [Multimodalna aplikacja edukacyjna wykorzystująca interaktywną animowaną postać]
PublicationThe aim of this study is to design a computer application that may assist teachers and therapists in multimodal manner in their work with impaired or disabled children. The application can be operated in many different ways, giving to a child with special educational needs a possibility to learn and train many skills or treat speech disorders. The main stress in this research is on the creation of animated character that will serve...
-
Cross-domain applications of multimodal human-computer interfaces
PublicationDeveloped multimodal interfaces for education applications and for disabled people are presented, including interactive electronic whiteboard based on video image analysis, application for controlling computers with mouth gestures and audio interface for speech stretching for hearing impaired and stuttering people and intelligent pen allowing for diagnosing and ameliorating developmental dyslexia. The eye-gaze tracking system named...
-
An extension to the FEEDB Multimodal Database of Facial Expressions and Emotions
PublicationFEEDB is a multimodal database that contains recordings of people expressing different emotions, captured by using a Microsoft Kinect sensor. Data were originally provided in the device’s proprietary format (XED), requiring both the Microsoft Kinect Studio application and a Kinect sensor attached to the system to use the files. In this paper, we present an extension of the database. For a selection of recordings, we also provide...
-
An audio-visual corpus for multimodal automatic speech recognition
Publicationreview of available audio-visual speech corpora and a description of a new multimodal corpus of English speech recordings is provided. The new corpus containing 31 hours of recordings was created specifically to assist audio-visual speech recognition systems (AVSR) development. The database related to the corpus includes high-resolution, high-framerate stereoscopic video streams from RGB cameras, depth imaging stream utilizing Time-of-Flight...
-
Scent emitting multimodal computer interface for learning enhancement
PublicationKomputerowy interfejs aromatyczny stanowi ważne uzupełnienie procesu stymulacji polisensorycznej. Stymulacja ta odgrywa kluczową rolę w terapii i kształceniu dzieci z zaburzeniami rozwoju (np. w przypadku autyzmu czy ADHD). Opracowany interfejs może stać się elementem wyposażenia tzw. sal doświadczania świata, ale może być także stosowany niezależnie stanowiąc znaczące wzbogacenie komputerowych programów edukacyjnych. Dzięki możliwości...
-
Evaluation of Decision Fusion Methods for Multimodal Biometrics in the Banking Application
PublicationAn evaluation of decision fusion methods based on Dempster-Shafer Theory (DST) and its modifications is presented in the article, studied over real biometric data from the engineered multimodal banking client verification system. First, the approaches for multimodal biometric data fusion for verification are explained. Then the proposed implementation of comparison scores fusion is presented, including details on the application...
-
Multimodal system for diagnosis and polysensory stimulation of subjects with communication disorders
PublicationAn experimental multimodal system, designed for polysensory diagnosis and stimulation of persons with impaired communication skills or even non-communicative subjects is presented. The user interface includes an eye tracking device and the EEG monitoring of the subject. Furthermore, the system consists of a device for objective hearing testing and an autostereoscopic projection system designed to stimulate subjects through their...
-
Performance Analysis of Developed Multimodal Biometric Identity Verification System
PublicationThe bank client identity verification system developed in the course of the IDENT project is presented. The total number of five biometric modalities including: dynamic handwritten signature proofing, voice recognition, face image verification, face contour extraction and hand blood vessels distribution comparison have been developed and studied. The experimental data were acquired employing multiple biometric sensors installed...
-
Multimodal coupling matrix for an array of rectangular slots on conducting cylinder
PublicationArtykuł prezentuje metodę wyznaczania sprzężeń wzajemnych pomiędzy aperturami promieniującymi położonymi na przewodzącym cylindrze. Pokazano sposób wyznaczania wielorodzajowej macierzy rozproszenia reprezentującej sprzężenia własne i wzajemne w badanej strukturze.
-
Lip movement and gesture recognition for a multimodal human-computer interface
Publication -
Pilot Testing of Developed Multimodal Biometric Identity Verification System
PublicationThe bank client identity verification system developed in the course of the IDENT project is presented. The total number of five biometric modalities including: dynamic signature proofing, voice recognition, face image verification, face contour extraction and hand blood vessels distribution comparison have been developed and studied. The experimental data were acquired employing multiple biometric sensors installed at engineered...
-
Testing the Accuracy of the Modified ICP Algorithm with Multimodal Weighting Factors
Publication -
Multimodal human-computer interfaces based on advanced video and audio analysis
PublicationMultimodal interfaces development history is reviewed briefly in the introduction. Some applications of multimodal interfaces to education software for disabled people are presented. One of them, the LipMouse is a novel, vision-based human-computer interface that tracks user’s lip movements and detect lips gestures. A new approach to diagnosing Parkinson’s disease is also shown. The progression of the disease can be measured employing...
-
Multimodal human-computer interfaces based on advanced video and audio analysis
PublicationMultimodal interfaces development history is reviewed briefly in the introduction. Examples of applications of multimodal interfaces to education software and for the disabled people are presented, including interactive electronic whiteboard based on video image analysis, application for controlling computers with mouth gestures and the audio interface for speech stretching for hearing impaired and stuttering people. The Smart...
-
Consciousness Study of Subjects with Unresponsive Wakefulness Syndrome Employing Multimodal Interfaces
PublicationThe paper presents a novel multimodal-based methodology for consciousness study of individuals with unresponsive wakefulness syndrome. Two interfaces were employed in the experiments: eye gaze tracking system – CyberEye developed at the Multimedia Systems Department, and EEG device with electrode placement in the international 10-20 standard. It was a pilot study for checking if it is possible to determine objective methods based...
-
The project IDENT: Multimodal biometric system for bank client identity verification
PublicationBiometric identity verification methods are implemented inside a real banking environment comprising: dynamic handwritten signature verification, face recognition, bank cli-ent voice recognition and hand vein distribution verification. A secure communication system based on an intra-bank client-server architecture was designed for this purpose. Hitherto achieved progress within the project is reported in this paper with a focus...
-
Analysis of results of large-scale multimodal biometric identity verification experiment
PublicationAn analysis of a large set of biometric data obtained during the enrolment and the verification phase in an experimental biometric system installed in bank branches is presented. Subjective opinions of bank clients and of bank tellers were also surveyed concerning the studied biometric methods in order to discover and to explore relations emerging from the obtained multimodal dataset. First, data acquisition and identity verification...
-
Processing of acoustical data in a multimodal bank operating room surveillance system
PublicationAn automatic surveillance system capable of detecting, classifying and localizing acoustic events in a bank operating room is presented. Algorithms for detection and classification of abnormal acoustic events, such as screams or gunshots are introduced. Two types of detectors are employed to detect impulsive sounds and vocal activity. A Support Vector Machine (SVM) classifier is used to discern between the different classes of...
-
Smart Pen - new multimodal computer control tool for graphomotorical therapy
PublicationW sytuacji, gdy około 15% populacji uczniów wykazuje cechy dyslektyczne, koniecznością staje się wyposażenie szkół w efektywne narzędzia do diagnozy i terapii tego rodzaju zaburzeń. Dzięki wykorzystaniu tabletu i specjalnie skonstruowanego długopisu wyposażonego w czujniki nacisku uzyskano możliwość monitorowania wielu parametrów, które do tej pory były dla terapeutów całkowicie niedostępne (np. pomiar nacisku na podłoże czy ścisku...
-
Multimodal Approach For Polysensory Stimulation And Diagnosis Of Subjects With Severe Communication Disorders
Publicationis evaluated on 9 patients, data analysis methods are described, and experiments of correlating Glasgow Coma Scale with extracted features describing subjects performance in therapeutic exercises exploiting EEG and eyetracker are presented. Performance metrics are proposed, and k-means clusters used to define concepts for mental states related to EEG and eyetracking activity. Finally, it is shown that the strongest correlations...
-
Detection of People Swimming in Water Reservoirs with the Use of Multimodal Imaging and Machine Learning
PublicationEvery year in many countries, there are fatal unintentional drownings in different water reservoirs like swimming pools, lakes, seas, or oceans. The existing threats of this type require creating a method that could automatically supervise such places to increase the safety of bathers. This work aimed to create methods and prototype solutions for detecting people bathing in water reservoirs using a multimodal imaging system and...
-
Driver’s Condition Detection System Using Multimodal Imaging and Machine Learning Algorithms
PublicationTo this day, driver fatigue remains one of the most significant causes of road accidents. In this paper, a novel way of detecting and monitoring a driver’s physical state has been proposed. The goal of the system was to make use of multimodal imaging from RGB and thermal cameras working simultaneously to monitor the driver’s current condition. A custom dataset was created consisting of thermal and RGB video samples. Acquired data...
-
Validating data acquired with experimental multimodal biometric system installed in bank branches
PublicationAn experimental system was engineered and implemented in 100 copies inside a real banking environment comprising: dynamic handwritten signature verification, face recognition, bank client voice recognition and hand vein distribution verification. The main purpose of the presented research was to analyze questionnaire responses reflecting user opinions on: comfort, ergonomics, intuitiveness and other aspects of the biometric enrollment...
-
Estimation of Broadband Complex Permeability Using SIW Cavity-Based Multimodal Approach
PublicationIn this article, an attractive multimodal substrate integrated waveguide (SIW) based methodology is presented for the characterization of magnetic materials in the broadband microwave frequency. The proposed approach employs a modified feed under-coupled SIW cavity instead of conventional feed over-coupled multiple SIW cavities; it uses the modified closedform expression, developed from the first principle to consider the effect...
-
Multimodal Network Based Graphs of Primitives Storage Concept for Web Mining CBIR
Publication -
Moving object detection and tracking for the purpose of multimodal surveillance system in urban areas
PublicationBackground subtraction method based on mixture of Gaussians was employed to detect all regions in a video frame denoting moving objects. Kalman filters were used for establishing relations between the regions and real moving objects in a scene and for tracking them continuously. The objects were represented by rectangles. The objects coupling with adequate regions including the relation of many-to-many was studied experimentally...
-
Zeta potential, effective diameter and multimodal size distribution in oil/water emulsion
Publication -
Multimodal Genetic Algorithm with Phase Analysis to Solve Complex Equations of Electromagnetic Analysis
PublicationIn this contribution, a new genetic-algorithm-based method of finding roots and poles of a complex function of a complex variable is presented. The algorithm employs the phase analysis of the function to explore the complex plane with the use of the genetic algorithm. Hence, the candidate regions of root and pole occurrences are selected and verified with the use of discrete Cauchy's argument principle. The algorithm is evaluated...
-
Multimodal Particle Swarm Optimization with Phase Analysis to Solve Complex Equations of Electromagnetic Analysis
PublicationIn this paper, a new meta-heuristic method of finding roots and poles of a complex function of a complex variable is presented. The algorithm combines an efficient space exploration provided by the particle swarm optimization (PSO) and the classification of root and pole occurrences based on the phase analysis of the complex function. The method initially generates two uniformly distributed populations of particles on the complex...
-
Testing Stability of Digital Filters Using Multimodal Particle Swarm Optimization with Phase Analysis
PublicationIn this paper, a novel meta-heuristic method for evaluation of digital filter stability is presented. The proposed method is very general because it allows one to evaluate stability of systems whose characteristic equations are not based on polynomials. The method combines an efficient evolutionary algorithm represented by the particle swarm optimization and the phase analysis of a complex function in the characteristic equation....
-
Characterization of Multimodal Silicas Using TG/DTG/DTA, Q-TG, and DSC Methods
Publication -
Attention-Based Deep Learning System for Classification of Breast Lesions—Multimodal, Weakly Supervised Approach
PublicationBreast cancer is the most frequent female cancer, with a considerable disease burden and high mortality. Early diagnosis with screening mammography might be facilitated by automated systems supported by deep learning artificial intelligence. We propose a model based on a weakly supervised Clustering-constrained Attention Multiple Instance Learning (CLAM) classifier able to train under data scarcity effectively. We used a private...
-
Multimodal detection and analysis of a new type of advanced Heinz body-like aggregate (AHBA) and cytoskeleton deformation in human RBCs
Publication -
New methods for assessment and stimulation of non-communicative patients employing advanced multimodal HCI . Nowe metody oceny i stymulacji pacjentów niekomunikatywnych z wykorzystaniem zaawansowanych interfejsów multimodalnych człowiek-komputer
PublicationIn most cases of patients with locomotor system damage it is possible to find a solution to the medical problems originating from the injury. However, it is much more difficult to prevent cognitive and emotional impairments. Therefore, we believe that the technological support of therapists working with such patients on an everyday basis may be essential. We have acquired experience in designing and providing diagnostic and therapeutic...
-
International Conference on Multimodal Interaction (International Conference on Multimodal Interfaces)
Conferences -
Joint workshop on Multimodal Interaction and Related Machine Learning Algorithms (now ICMI-MLMI)
Conferences