Robust, Unsupervised, and Ubiquitous Pose and Object Information for Robot Action Recognition
Leveraging a novel attention mechanism, 3D convolutional neural networks, and pose data, we demonstrate how to boost human activity recognition in RGB data without the need for high-fidelity pose data. Results published in ICRA 2021.
Automatic target recognition and geo-location for side scan sonar imagery
Cross-disciplinary project with mechanical engineering team. Demonstrated that convolutional deep neural network (CNN) architectures could be used to find crashed aircraft black-boxes in underwater sonar imagery.
Dynamic Deep Learning
Led algorithm in group with diverse skill set to automate optimization of CNN architectures for image recognition under resource constraints Used Python, Keras, TensorFlow, SLURM, Matplotlib. Resulted in publication in CVPR 2018 and patent.
On the need for imagistic modeling in story understanding
There is ample evidence that human understanding of ordinary language relies in part on a rich capacity for imagistic mental modeling. We argue that genuine language understanding in machines will similarly require an imagistic modeling capacity enabling fast construction of instances of prototypical physical situations and events, whose participants are drawn from a wide variety of entity types, including animate agents
Finding Islands of Predictability in Action Forecasting
We address dense action forecasting, the problem of predicting future action sequence over long durations based on partial observation. Our key insight is that future action sequences are more accurately modeled with variable, rather than one, levels of abstraction, and that the optimal level of abstraction can be dynamically selected during the prediction process. Our experiments show that most parts of future action sequences can be predicted confidently in fine detail only in small segments of future frames, which are effectively ``islands'' of high model prediction confidence in a ``sea'' of uncertainty.
PLAN-B- Predicting Likely Alternative Next Best Sequences for Action Prediction
Action prediction focuses on anticipating actions before they happen. Recent works leverage probabilistic approaches to describe future uncertainties and sample future actions. However, these methods cannot easily find all alternative predictions, which are essential given the inherent unpredictability of the future, and current evaluation protocols do not measure a system's ability to find such alternatives. We re-examine action prediction in terms of its ability to predict not only the top predictions, but also top alternatives.