Robust, Unsupervised, and Ubiquitous Pose and Object Information for Robot Action Recognition
Robust, Unsupervised, and Ubiquitous Pose and Object Information for Robot Action Recognition

Leveraging a novel attention mechanism, 3D convolutional neural networks, and pose data, we demonstrate how to boost human activity recognition in RGB data without the need for high-fidelity pose data. Results published in ICRA 2021.

PytorchPython
Automatic target recognition and geo-location for side scan sonar imagery
Automatic target recognition and geo-location for side scan sonar imagery

Cross-disciplinary project with mechanical engineering team. Demonstrated that convolutional deep neural network (CNN) architectures could be used to find crashed aircraft black-boxes in underwater sonar imagery.

KerasTensorFlowPython
Deep learning for viral tropism detection
Deep learning for viral tropism detection

Led design and implementation of CNN classifier to identify viral tropism from protein sequences. Used Keras, Tensorflow, Python, Matplotlib, and SLURM. Achieved top prediction accuracy on real-world viral data. Results published in Health Security 2019.

KerasTensorFlowPython
Dynamic Deep Learning
Dynamic Deep Learning

Led algorithm in group with diverse skill set to automate optimization of CNN architectures for image recognition under resource constraints Used Python, Keras, TensorFlow, SLURM, Matplotlib. Resulted in publication in CVPR 2018 and patent.

KerasTensorFlowPythonSLURM
On the need for imagistic modeling in story understanding
On the need for imagistic modeling in story understanding

There is ample evidence that human understanding of ordinary language relies in part on a rich capacity for imagistic mental modeling. We argue that genuine language understanding in machines will similarly require an imagistic modeling capacity enabling fast construction of instances of prototypical physical situations and events, whose participants are drawn from a wide variety of entity types, including animate agents

BlenderPython
Finding Islands of Predictability in Action Forecasting
Finding Islands of Predictability in Action Forecasting

We address dense action forecasting, the problem of predicting future action sequence over long durations based on partial observation. Our key insight is that future action sequences are more accurately modeled with variable, rather than one, levels of abstraction, and that the optimal level of abstraction can be dynamically selected during the prediction process. Our experiments show that most parts of future action sequences can be predicted confidently in fine detail only in small segments of future frames, which are effectively ``islands'' of high model prediction confidence in a ``sea'' of uncertainty.

PytorchPythonSLURM
PLAN-B- Predicting Likely Alternative Next Best Sequences for Action Prediction
PLAN-B- Predicting Likely Alternative Next Best Sequences for Action Prediction

Action prediction focuses on anticipating actions before they happen. Recent works leverage probabilistic approaches to describe future uncertainties and sample future actions. However, these methods cannot easily find all alternative predictions, which are essential given the inherent unpredictability of the future, and current evaluation protocols do not measure a system's ability to find such alternatives. We re-examine action prediction in terms of its ability to predict not only the top predictions, but also top alternatives.

PytorchPythonSLURM