An ERP N400 study: Semantic processing across modalities in the human brain

Abstract

Semantic processing in the brain of both language and action has been associated with the N400; an event-related potential (ERP) that is typically present when information has violated one’s semantic expectations. We know that the brain receives and processes information from multiple modalities which means that cross-modal semantic processing is more aligned with how semantic processing is likely to occur in comparison to unimodal semantic processing. The N400 is a consistent effect across cross-modal language-based studies. But we know that the topographical distribution of the N400 does vary between language-based and non-language-based paradigms. Therefore, to investigate cross-modal semantic processing within the action domain, we presented participants with photographs portraying the implementation of common actions. These sequences concluded with a sound that was either congruent or incongruent with the prior action photographs. In our ERP study of 25 participants, with a mean age of 23 years (SD = 10.78 years), we¹ found an N400 effect for incongruent information processing. In addition, our findings showed a delayed N400 effect and a reduced P200 amplitude for incongruent information. These results suggest that cross-modal semantic processing of action sequences requires an increased cognitive workload which is evidenced when semantic processing does not progress as expected. Considered as a whole, these results indicate that cross-modal semantic processing is similar to unimodal processing. Cross-modal information likely requires the involvement of additional cognitive processes that are not present during unimodal paradigms.