The role of contextual modulations has been extensively studied in basic sensory and cognitive processes. However, little is known about their impact on social cognition, let alone their disruption in disorders compromising such a domain. In this chapter, we flesh out the social context network model (SCNM), a neuroscientific proposal devised to address the issue. In SCNM terms, social context effects rely on a fronto-temporo-insular network in charge of (a) updating context cues to make predictions, (b) consolidating context–target associative learning, and (c) coordinating internal and external milieus. First, we characterize various social cognition domains as context-dependent phenomena. Then, we review behavioral and neural evidence of social context impairments in behavioral variant frontotemporal dementia (bvFTD) and autism spectrum disorder (ASD), highlighting their relation with key SCNM hubs. Next, we show that other psychiatric and neurological conditions involve context-processing impairments following damage to the brain regions included in the model. Finally, we call for an ecological approach to social cognition assessment, moving beyond widespread abstract and decontextualized methods.