Intracellular processes such as cytoskeletal organization and organelle dynamics exhibit extensive subcellular heterogeneity. Although recent advances in fluorescence microscopy allow researchers to acquire an unprecedented amount of live cell image data at high spatiotemporal resolutions, the traditional ensemble-averaging of uncharacterized heterogeneity could mask important activities. Here, we establish an unsupervised machine learning framework called DeepHACKS (Deep phenotyping of Heterogeneous Activities in the Coordination of cytosKeleton at the Subcellular level) for “deep phenotyping,” which identifies rare subcellular phenotypes specifically sensitive to molecular and environmental perturbations. DeepHACKS dissects the heterogeneity of subcellular time-series datasets by allowing bi-directional LSTM (Long-Short Term Memory) neural networks to extract fine-grained temporal features by integrating autoencoders with the prior information from traditional machine learning outcomes. We applied DeepHACKS to subcellular protrusion dynamics in pharmacologically perturbed epithelial cells by CK666, Cytochalasin D, and blebbistatin. DeepHACKS characterized fine differential responses of leading edge dynamics specific to each perturbation and revealed the fine-grained subcellular and single-cell phenotypes driven by “bursting” and “accelerating” protrusions. This suggests that the temporal features directly learned from live cell images enabled us to identify drug-related subcellular and single-cell phenotypes. DeepHACKS provides an analytical framework for detailed and quantitative understandings of molecular mechanisms hidden in temporal heterogeneity and can be potentially applied to live cell-based drug discovery and diagnostics.