Журнал высшей нервной деятельности им. И.П. Павлова, 2019, T. 69, № 2, стр. 175-186

IT IS NOT WHAT YOU THINK IT IS: ERP CORRELATES OF VERBAL AND NON-VERBAL AMBIGUITY PROCESSING

M. G. Filippova 1*, O. V. Shcherbakova 1, Y. Y. Shtyrov 12

1 Department of Psychology, Saint Petersburg State University
Saint Petersburg, Russia

b Center of Functionally Integrative Neuroscience, Aarhus University
Aarhus, Denmark

* E-mail: m.g.filippova@spbu.ru

Поступила в редакцию 24.11.2017
После доработки 20.03.2018
Принята к публикации 14.05.2018

Полный текст (PDF)

Аннотация

Perceptual information is often ambiguous and we have to deal with such ambiguity to ensure optimal behavior; yet, the mechanisms that our brain uses for processing ambiguous stimuli are not well understood. In the current study, we tested whether there were any common markers of neural processing of ambiguity, regardless of its type. To this end, ERPs (event-related potentials) were elicited under similar experimental conditions by either verbal or non-verbal information: ambiguous figures vs. verbal jokes. It has been suggested that ambiguous graphical information triggers a mismatch conflict at earlier stages of processing, whereas in case of perception of ambiguous written texts, it takes place at later stages, associated with semantic analysis. Results of our experiment show that perception of both ambiguous figures and verbal jokes was related to semantic reversion, as the amplitude of the negative-going N400 component increased in response to both pictorial and verbal stimuli that were correctly identified as having more than one meaning, in contrast to otherwise similar but unambiguous control stimuli.

Keywords: ambiguity perception, event-related potentials (ERP), N400, N200, brain, figure processing, jokes comprehension, humor

INTRODUCTION

Ambiguity is ambivalent: on one hand, we tend to reduce ambiguous information to a single meaning and, therefore, make it feel more certain and predictable. For example, when a student is listening to a lecturer and doubting whether his/her statements should be interpreted in either one way or another (depending on a background, situation, lecturer’s mood, current social context, etc.), it is very likely that the student will ask clarification questions and thus figure out what the exact meaning is. But, on the other hand, we value ambiguity as an essential component of any literature, art or everyday communication as long as it provides us with enjoyable interplay of meanings and brings us to new semantic contexts. In both cases, our minds and brains need to detect a certain piece of information as being ambiguous and resolve the ambiguity. The question is what the cognitive mechanisms underlying this detection are and how the brain implements them.

Ambiguous figures are a special case of ambiguity. Ambiguous figures combine two alternative images within the same picture and, thus, provide an observer with a necessity to make a perceptual choice between two alternative meanings (fig. 1 (a, b)). However, most people are able to recognize both meanings of such ambiguous figures. A sudden recognition of an alternative meaning of ambiguous figure is called perceptive reversal. Some authors argue that reversals of ambiguous figures can be explained by an assembly of stimulus elements into object ‘gestalts’ that takes place early in the course of visual processing and does not require any processing at higher level [Intaité et al., 2013; Kornmeier, Bach, 2014]. This idea is supported by the fact that working memory load does not affect the perceptual reversal speed [Intaité et al., 2014]. This means that the reversal probably takes place at lower levels that do not involve memory processes. Obviously, these processes precede the recognition of the second meaning of ambiguous figure; the role of later or higher-level processes that lead to recognition of an alternative meaning remains unclear.

Fig. 1.

Example of non-verbal stimuli (а, b – am-biguous; c, d – non-ambiguous).

Рис. 1. Пример невербальных стимулов (а, b – многозначные; c, d – однозначные).

It has been reported that ambiguity conflicts take place during the processing of stimulus elements roughly 130 ms after stimulus onset, and it takes about 350 ms to switch from the assembly of stimulus elements into the object ‘gestalt’ to the indication of perceptual reversal; this was, for instance, shown for both Necker’s cube [Intaité et al., 2013] and Boring’s old/young woman [Kornmeier, Bach, 2014] (fig. 1 (a)). Kornmeier and Bach [2014] also point out that 350 ms is a rather long period of time in the visual processing time scale and highlight the need to scrutinize the processes that take place within this time window and their potential links to perceptual awareness.

Gao and colleagues [Gao et al., 2015] discuss the difficulty of differentiating perceptual processing from semantic one. Their results showed that alteration of simple perceptual characteristics (such as colors) of abstract visual stimuli can dramatically alter their meanings and change behavioral and neural correlates of memory. Probably, in the context of geometrical stimuli processing (such as Necker’s cube), transformations can only take place at an early perceptual level. However, it is hard to see the switch from one meaning to another with no involvement of semantic levels when it comes to more complex figures (such as in fig. 1 (a, b)). That is, perceptual reversal between different meanings of such complex figures may require more than simple visual transformation based on changes in their spatial configuration; namely, it should require changes in the semantic content of the stimulus.

ERP CORRELATES OF SEMANTIC AND NON-SEMANTIC PROCESSING OF AMBIGUOUS STIMULI

ERP correlates of non-semantic processing of ambiguous stimuli

Through the use of Necker’s cube, two main ERP components corresponding to perceptual reversals of ambiguous figures have been suggested: Reversal Positivity and Reversal Negativity [Intaité et al., 2013; Kornmeier, Bach, 2014]. Reversal Positivity occurs around 100–130 ms, Reversal Negativity – around 200–260 ms after stimulus onset, preceding the behavioral recognition of an alternative meaning. Sometimes later positive components are also considered, although not all of these effects are always observed within a single experiment.

Another widely discussed correlate of non-semantic ambiguity processing that is related to perceptual mismatch is N200. N200 is a negative ERP component emerging at fronto-central electrode locations approximately 200 ms after stimulus onset. This component is common for response-incongruent tasks [Van Veen, Carter, 2002; Renn, Cote, 2013], and is reported to depend on stimulus probability [Nieuwenhuis et al., 2003; Enriquez-Geppert et al., 2010].

According to Gajewski and Falkenstein [Gajewski, Falkenstein, 2013], N200 is caused by the task difficulty that depends on stimulus probability, target-distractor perceptual similarity and sensory mismatch. Enhancement of N200 amplitude is often interpreted as a concomitant increase of the depth of encoding process [Kubota, Ito, 2007], and its decrease is viewed as reflecting lower perceptual synthesis intensity [Morozova et al., 2012].

In an impressive body of literature, N200 is often associated with conflict-related and evaluative processing stages [e.g., Botvinick et al., 2004; Huster et al., 2013]. Some authors put special emphasis on the link between N200 and cognitive control processes [Kopp et al., 1996]; in particular, N200 is considered a marker of conflict detection [Van Veen, Carter, 2002; Nieuwenhuis et al., 2003; Siemann et al., 2016], including inhibition of a prepotent response. Observations (e.g., [Frings, Groh-Bordin, 2007]) of an enhanced N200 in ignored repetition (IR) trials (when incongruent trials display greater and more negative activation than neutral trials) suggest correspondence between this component and the inhibition mechanism. Accordingly, fronto-central N200 is often interpreted in terms of response conflict or response inhibition, which implies that stop-reaction and conflict evaluation might be generated by the same brain region [e.g., Iannaccone et al., 2015].

It is assumed that N200 has no relation to semantic processing of stimuli [e.g. Ortells et al., 2015; Du et al., 2014]. Some researchers, however, report the increase of N200 amplitude under experimental conditions requiring some semantic assessment. For instance, Lei and colleagues [Lei et al., 2014] studied decision-making under conditions of risk and ambiguity. They found an enhancement of N200 amplitude evoked by the decision-making under ambiguous condition compared to risk condition. The latency and polarity of N200 clearly overlaps with the Reversal Negativity mentioned above, although the exact relationship between the two still remains to be explored.

ERP correlates of semantic processing of ambiguous stimuli

N400 is thought to be the key correlate of semantic ambiguity detection. The N400 is a negative-going deflection of evoked potentials observed over central scalp areas, peaking around 400 ms after the stimulus onset. Usually N400 emerges within a time window of 250–500 ms as a brain response to words and other important (or potentially important) stimuli presented visually or auditorily [Kutas, Federmeier, 2000].

N400 is related to general semantic processes and has been linked to activation of memory systems, information structuring and semantic integration [Coulson et al., 2005; Friedrich, Friederici, 2006]. Larger, more negative N400 activation corresponds to semantic anomalies. Amplitude of N400 is often viewed as an index of difficulty in accessing/reactivating representations that depends on the lack of essential context information [Wieser, Wieser, 2003; Friedrich, Friederici, 2006].

In neuroscience of language, N400 is often linked to semantic integration of words into the context of a sentence [e.g., Coulson et al., 2005; Özyürek, 2014]. Kutas and Hillyard [Kutas, Hillyard, 1980] showed that N400 is enhanced when a word occurs in an unexpected or unrelated context. It is currently believed that N400 amplitude is rather modulated by predictability than by the exact semantic content of the context [Lau et al., 2013]. This notion is supported by some studies that report N400 in response to semantically inconsistent objects appearing within apparently meaningless synthesized scenes [Vo et al., 2016].

Most researchers recognize N400 as an indicator of semantic processing, even for non-verbal stimuli, emphasizing the importance of differentiation between meaningful and meaningless objects (with smaller N400 for meaningful ones [Soldan et al., 2010]) even in cases when meaning is not an objective stimulus parameter but is assigned by the recipient himself/herself (see, e.g., experiments with abstract visual shapes [Gao et al., 2015]. Another interesting result was obtained in the study by Sanguinetti and colleagues [Sanguinetti et al., 2014], where N400 was detected in response to silhouettes camouflaged within the ground sides of stimulus objects that recipients were unaware of. The authors consider this finding to be a neurophysiological evidence that semantic access can even occur for the apparently shapeless ground side of a border.

It has been shown that ambiguous words (homonyms and polysemes) elicit larger N400 amplitudes than unambiguous words [Haro et al., 2017]. Larger N400 amplitudes have been reported for words with many semantic features or associates than for those with few semantic features or associates [Rabovsky et al., 2012], and also for words that appear in a wide range of diverse contexts as opposed to a restricted set of contexts [Hoffman et al., 2013]. All these ambiguous words are usually described with an umbrella term “words with high semantic richness” [Vergara-Martínez et al., 2017]. Usually, semantically richer words produce more negativity compared to words with less semantic richness, however this regularity is affected by the particular task requirements [Amsel, Cree, 2013]. Increase of N400 amplitude was also reported in response to sarcastic phrases [Gibson et al., 2015], humor [Feng et al., 2014] and garden-path jokes whose meaning is ambiguous until a certain point [Mayerhofer, 2015; Mayerhofer, Schacht, 2015]. This research points to the N400 as another potential biomarker of ambiguity processing in the brain.

THE PRESENT STUDY

The aim of the present study was to test experimentally whether there are common ERP correlates of processing ambiguous stimuli of rather different nature: verbal texts and non-verbal graphical figures. We have not found any evidence of such comparisons in the existing literature on ambiguity perception. As stimuli, we used so-called “canned” jokes whose meaning changes as the story unfolds, and ambiguous figures that can be perceived as two alternative graphical objects.

We examined both earlier and later ERP components, focusing particularly on those that have been linked to incongruence and conflict detection: N200 and N400. Considering the characteristics of N200 and N400, we predicted that they were likely to be generated in response to ambiguous stimuli that require participants to figure out alternative interpretations. Both N200 and N400 refer to detection of a mismatch: N200 – at the perceptual level, N400 – at the semantic one.

Based on the previous research, we expected to see an increase of N400 amplitude during successful comprehension of humor, and an increase of N200 (or even earlier components, such as reversal positivity) amplitude when detecting both interpretations of ambiguous figures. As far as, in our view, changes in semantic content are intrinsic to both considered cases of ambiguity, we also expect to register N400 during the perceptual reversals of ambiguous figures.

METHOD

Participants

22 volunteers (mean age 23 y.o., 16 females) were recruited through the Laboratory of Psychophysiology of Saint Petersburg State University. All of them were healthy right-handed native speakers of Russian and had normal or corrected to normal vision. All the participants were treated in accordance with the Declaration of Helsinki and provided informed consent prior to the participation in the experiment. Data from two participants were excluded from the analysis as they had more than 20% of incorrect responses due to low compliance with the task.

Procedure

Participants went through two similar experimental procedures with 2 types of stimuli: 1) 36 ambiguous (matching two different semantic categories) and 36 non-ambiguous figures; 2) 14 verbal jokes and 14 similar but non-humoristic short stories. In the first block of the experiment, participants were presented with figures of both types. The task was to identify whether each figure was ambiguous or non-ambiguous. In the second block, participants were presented with verbal stories and the task was to identify whether each story was a joke or not.

There was no feedback on the response correctness/incorrectness. Response time was not limited. Experiment lasted approximately 30 minutes.

Materials

Non-verbal stimuli (ambiguous figures)

We used 36 ambiguous figures as non-verbal stimuli. A figure is considered ambiguous if it can be categorized/interpreted in two different ways at the same time (fig. 1 (a, b)). 36 non-ambiguous figures portrayed in the same technique as ambiguous figures, and matching the latter ones in terms of presentation were used as control stimuli (see fig. 1 (c, d)).

The figures (10 × 10 cm) were presented one by one in the central part of the computer screen. The participant’s task was to tell whether the figure presented had one or two meanings by pressing “1” or “2” on the computer keyboard. Each figure was presented for 500 ms only in order to prevent a participant from perusing it. Response time was not limited and there was no feedback on the response correctness/incorrectness. In a short time (randomly varied from 900 to 1500 ms) after each response, the next figure was presented. In total, each participant had to estimate 72 figures.

Verbal stimuli (“canned” jokes)

As ambiguous verbal stimuli, we used the so-called “canned” jokes. These jokes are short funny stories ending with a punch line [Martin, 2010]. “Canned” jokes usually contain some ambiguity because one might take them either in humoristic or non-humoristic way. Within cognitive approaches to humor [Attardo, Raskin, 1991; Wyer, Collins, 1992], the latter is considered a result of two contradictive schemes, frames or scenarios colliding with each other within the same situation. One scenario usually corresponds to a normal, predictable and the most expected sequence of events, while another one contradicts that and corresponds to an alternative and unpredictable outcome. In case of verbal jokes, these two scenarios meet at a keyword of phrase, which is usually described as a joke’s “point” or “punch line”. If the point of a joke is taken by a reader or listener, both scenarios activate and the text is perceived as an ambiguous one. On the contrary, if the point is not taken, one of the two scenarios does not activate and the text remains non-ambiguous to the reader/listener. Thus, verbal jokes can be viewed as a special case of verbal ambiguous stimuli (amongst metaphors, fables, proverbs, etc.) and are a good model for studying perception of ambiguity.

We used 14 such jokes with no obvious religious, sexual or political connotations (see tab. 1 for an example). 14 similar but non-humoristic short stories were used as control stimuli; they were composed by modifying the punch lines of the jokes in order to make them look as descriptions of normal, non-funny everyday activities. All the control non-humoristic verbal stories matched the jokes in terms of length as can be seen in tab. 1.

Table 1.

An example of verbal “canned” joke with humoristic (original) and non-humoristic (altered) endings Таблица 1. Пример шутки с юмористическим (оригинальным) и неюмористическим (измененным) окончаниями

  Joke Control unambiguous stimulus
Main text Two good friends are chatting. One of them asks:
– So, dude, that grand quarrel with your girlfriend – what did it end up with?
– Well, she came to me on her knees.
– Really?! And what did she say?
Ending – She said: “Ok, you can clamber out from underneath the bed already, I am not angry with you anymore”. – She said: “Please, forgive me. I am so sorry for behaving so badly!”

These written stimuli were presented centrally on the screen. First, the main text, with the last disambiguating sentence missing, was presented at once and, after a participant pressed ENTER key, followed by presentation of the last key phrase (which could be either a joke’s punch line or the final sentence of a non-humoristic story, correspondingly). This last key phrase was presented in a word-by-word format, to enable time-locking of brain responses to the critical information. On average, last key phrases were as long as 5 words. A participant’s task was to tell whether the text presented at the screen was a joke or not by pressing “1” or “2” on the computer keyboard. Each word of the last sentence was presented for 500 ms. Response time was not limited and no feedback on the response correctness was given. In a short time (randomly varied from 900 to 1500 ms) after the response, the next text was presented. Each participant had to estimate 28 verbal texts within 2 experimental series: for counterbalancing, the texts which served as jokes in the Series 1, were replaced with their non-humoristic analogues in the Series 2, and vice versa. All the texts were presented in Russian.

Apparatus

Subjects were seated in a dimly lit, sound-attenuated room at a distance of 85 cm from a 17-inch monitor (SyncMaster 171T CRT [Samsung Group, Seoul, South Korea], resolution 1024 × 768 pixel, 75 Hz vertical refresh). Stimulus presentation was performed using ExperStim v.3.3 software (Laboratory of Psychophysiology, Saint Petersburg State University). Stimulus presentation was synchronized with the screen refresh rate of 100 Hz. Manual responses were collected using a keyboard.

EEG recording and data analysis

During the experiment, the subject’s electroencephalogram (EEG) and electrooculogram (EOG) were recorded using 21 Ag/AgCl electrodes. Nineteen electrodes were mounted in an elastic electrode cap at FP1, FP2, F7, F3, Fz, F4, F8, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1 and O2 locations of the international 10/20 system. Two EOG electrodes were placed to the outer canthi and the infraorbital ridge of the right eye. Linked reference electrodes were mounted at the left and right earlobes. Position of the body in the armchair and the head position were adjusted to avoid muscle artifacts from the neck muscles. Impedances were kept below 20 kOm. An additional electrode at FPz position was used as a ground electrode site. Signals were amplified with a Telepath 104P EEG amplifier (Potential LLC, Saint Petersburg, Russia) using a bandpass of 1.6–70 Hz and a sampling rate of 250 Hz. 50 Hz notch filter was applied to eliminate line noise.

Continuous EEG data were epoched into segments starting 200 ms before and ending 700 ms after stimulus presentation, that is 900-ms interval in total. Eye-movement artifact correction was applied by oculogram subtraction (using linear regression algorithm, as implemented in WinEEG software, Mitsar, Saint Petersburg, Russia). Baseline correction of -200 to 0 ms pre-stimulus interval was applied. Trials with incorrect responses were excluded from further statistical analysis. 2 out of 24 participants had more than 20% of incorrect responses, so their data discarded.

ERP voltages that were more than 3 standard deviations above or below the group mean were excluded from further analysis as outliers (these accounted for 1.37% for verbal and 1.11% for non-verbal trials). The resulting clean epochs were averaged to produce subject- and condition-specific ERPs.

Statistical analysis of the ERP effects was done using repeated measures ANOVA with mean amplitude values computed for each participant and each electrode in N200 and N400 time windows. Considering the topological specificity of N200 and N400, we focused on the analysis of potentials from anterior and central electrode sites. Average ERP amplitudes were extracted for each subject, condition and stimulus type. ERP amplitudes were averaged across 50-ms wide time bins which were then submitted to statistics. The amplitudes of N200 and N400 ERP components were dependent variables. Any topographic effects were analyzed by dividing the electrode array into 6 regions of interest (ROIs) arranged in a 2 × 3 fashion using factors caudality (anterior vs. central) and laterality (left, middle, right). Electrode distribution in these ROIs were as follows: left anterior (Fp1, F3, F7), middle anterior (Fz), right anterior (Fp2, F4, F8), left central (Т3, С3), middle central (Cz), right central (Т4, C4).

Thus, for each stimulus type (verbal and non-verbal) within both components (N200, N400), the influence of 3 factors was analyzed by entering response amplitudes into a 3-way ANOVA: 2 (conditions: ambiguity or non-ambiguity) × × 3 (laterality: left, middle, right) × 2 (caudality: anterior, central).

RESULTS

Visual inspection of group average ERP waveforms showed that both N200 and N400 components emerged in response to the stimuli of both (verbal and non-verbal) types (see fig. 2 and 3). In our experiment, N200 peaked somewhat earlier than the classic N200 interval: in the time window of 100–200 ms after the stimulus onset in all considered ROIs, while the N400 was manifested in its usual time window of 250–500 ms.

Fig. 2.

Grand-averaged ERP waveforms for ambiguous and for unambiguous conditions at 6 ROIs for non-verbal stimuli. Negative values are displayed downwards.

Рис. 2. Усредненные по группе испытуемых ВП в многозначных и однозначных условиях в 6 областях интересов для невербальных стимулов. Отрицательные значения отображаются снизу.

Fig. 3.

Grand-averaged ERP waveforms for ambiguous and for unambiguous conditions at 6 ROIs for verbal stimuli. Negative values are displayed downwards.

Рис. 3. Усредненные по группе испытуемых ВП в многозначных и однозначных условиях в 6 областях интересов для вербальных стимулов. Отрицательные значения отображаются снизу.

3.1 Non-verbal stimuli

Figure 2 illustrates ERP waveforms for figures in different conditions at the six ROIs for N200 and N400 time window.

For the N200 component, only the main effect of caudality (F(1, 516) = 5.86, p = 0.016) was found to be significant, reflecting centro-posterior distribution of this response. Neither the main effect of laterality (p = 0.842) nor that of condition (p = 0.110), the latter being of principal interest, reached significance. Notably, the effect of condition did not show significant differences in any of the 6 ROIs analyzed (all р-values > 0.1). Also, no statistically significant interactions between any of the factors were observed.

The same ANOVA as above was performed on the mean amplitude of the 250–500 ms time window, for the N400 component. Here, we found a main effect of condition (F(1, 516) = 18.31, p < < 0.001), which was driven by a larger ERP amplitude in ambiguity rather than non-ambiguity condition. A main effect of caudality (F(1, 516) = = 53.51, p < .001) was also found to be significant, reflecting a stronger response at anterior locations than at the more posterior ones. No significant interactions were identified. Perception of ambiguous figures was followed by greater negativity rather than that of non-ambiguous figures in each of 6 considered ROIs. Moreover, as it can be seen in fig. 2, the greatest difference between experimental and control conditions was found (1) in midline cluster (Cz and Fz positions), in line with a common N400 topography, and (2) over the right hemisphere.

Tab. 2 presents amplitude differences between ambiguous and non-ambiguous conditions in all clusters within the time window of 250–500 ms.

Table 2.

Mean N400 amplitude and amplitude differences between ambiguous and non-ambiguous conditions for verbal and non-verbal stimuli at three levels of the laterality factor Таблица 2. Средние значения амплитуды N400 и разность амплитуды между многозначными и однозначными условиями для вербальных и невербальных стимулов по трем градациям фактора “латеральность”

  Nonverbal stimuli Verbal stimuli
M
ambiguity
M
non-ambiguity
D M ambiguity M
non-ambiguity
D
Left –.580 .284 0.864* 1.185 2.196 1.012**
Middle –1.097 0.256 1.353* 1.563 2.731 1.168*
Right –0.312 0.593 0.905** 1.592 2.375 0.783*

Note: Post Hoc comparisons obtained using Tukey’s HSD test. M – mean; D – amplitude difference between mean ambiguity and mean non-ambiguity conditions, * – p < 0.05; ** – p < 0.01.

In all the three levels of the left-right laterality distribution, differences between experimental and control conditions were significant: greater negativity corresponds to ambiguous rather than non-ambiguous condition. However, as seen in tab. 2, for non-verbal stimuli the greatest differences in magnitude between these conditions appeared over the midline (F(1, 84) = 5.39, p = = 0.023), whereas the greatest differences in terms of significance level appeared over the right hemisphere (F(1, 216) = 7.15, p = 0.008).

Thus, ambiguous figures were found to elicit stronger N400 deflection than non-ambiguous ones. Notably, this difference is more obvious at the midline and over the right scalp locations. At the same time, no significant difference between ambiguous and non-ambiguous figures was found in the N200 time window.

Verbal stimuli

Similar to non-verbal stimuli, the N200 negative waveform emerged within the time window of 100–200 ms and was well-defined at both anterior and central sites. However, no significant main effects or interactions were found for the N200 component. There were no differences between N200 in experimental and control conditions in any of the considered ROIs.

For N400 component, the main effects of condition (F(1, 516) = 15.50, p < 0.001) and caudality (F(1, 516) = 9.0, p = 0.003) were found to be significant. No significant interaction effects were observed for this component.

Similar to the graphical stimuli, N400 (in the time window of 250–500 ms) to texts was found to be increased in ambiguous condition compared to non-ambiguous one. In other words, presentation of verbal jokes was followed by an increase of N400 amplitude as opposed to control non-humoristic verbal stories. This difference was statistically significant for all considered ROIs – although, as it can be seen in fig. 3, mostly for the electrodes on the midline (F(1, 84) = 4.64, p = = 0.034) and over the left hemisphere (F(1, 216) = = 8.56, p = 0.004). The differences are presented in tab. 2: their magnitudes are higher in the middle ROI (the same as for figures) and their statistical significance is greater in the left one, as could be expected for verbal stimuli.

Thus, verbal jokes were found to elicit stronger N400 deflection than non-ambiguous control texts (mostly in the middle and left electrode locations), whereas no difference between ambiguous and non-ambiguous texts was found in the N200 time window analysis.

DISCUSSION OF RESULTS

Results obtained in previous studies suggest that, in case of ambiguous figures perception, ambiguity conflict takes place at a very basic processing level [Intaité et al., 2013; Kornmeier, Bach, 2014], whereas in case of ambiguous verbal texts, this conflict emerges at a higher level related to semantic processing [Kutas, Federmeier, 2000; Gibson et al., 2015; Vergara-Martínez et al., 2017; Haro et al., 2017]. In earlier studies of EEG correlates of ambiguous visual perception, it was assumed that perceptual reversion was related to earlier, non-semantic mechanisms [Intaité et al., 2013]. Cognitive mechanisms at a higher level, activating after perceptual reversion of an ambiguous figure and before any awareness of this reversion appears (350 ms approximately), have, however, remained unclear [Kornmeier, Bach, 2014].

We predicted that during perception of more complex ambiguous figures (not just simple shapes), cognitive processes might be similar to those specific for stimuli with high semantic richness [Vergara-Martínez et al., 2017]. In particular, we expected to find similarity in perception of complex ambiguous figures and verbal jokes presented under same experimental conditions. For that reason, we focused on searching for potential correlates of perception of ambiguous figures and verbal jokes, rather than on registering early visual components corresponding to figure perception. We considered ERP correlates both related and unrelated to semantic processing, paying special attention to the N200 and N400 components. N200 is an earlier component that is traditionally associated with perceptual processing of stimuli [e.g. Ortells et al., 2015; Du et al., 2014]; N400 is believed to be related to semantic prediction and comprehension, meaning processing and context integration [e.g. Kutas, Federmeier, 2000; Coulson et al., 2005; Friedrich, Friederici, 2006].

Based on the evidence obtained in previous studies, we predicted that both N200 and N400 components could index alternative meaning detection. We did not focus on later ERP components that are also thought to be related to ambiguity processing – such as P600 which correlates with secondary top-down monitoring processes [Kirsten et al., 2014]. Results of our experiment showed similar change in ERPs elicited by perception of ambiguous figures and verbal jokes. This change was the increase of N400 in response to ambiguous figures or verbal jokes as compared to presentation of non-ambiguous control figures and non-humoristic verbal stories. The obtained results suggest that the ambiguity in both graphical figures and verbal texts contributes to an increased intensity of information processing, which, however, does not occur at the perceptual integration stage (this would be reflected by an increase of N200 amplitude, which we did not observe) but takes place at the later semantic analysis stage and is manifested as an increase of the N400 negativity.

Our findings are in line with those reported previously by Kutas and colleagues (e.g., [Federmeier, Kutas, 2001]), showing that a stimulus presented within an unexpected context is followed by the increase of N400 component. In our experiment, the changes in N400 (that is often considered an indicator of memory search success and semantic context integration) are a manifestation of the increased difficulty of semantic and associative comprehension processes under ambiguous conditions. According to Kutas and colleagues [Kutas, Hillyard, 1980; Federmeier, Kutas, 2001], the increase of N400 might be a result of inability to automatically process the input, triggering an additional, more laborious processing that requires top-down control and voluntary attention. This reverification can be a result of successful actualization of an alternative meaning while perceiving ambiguous figures and verbal jokes. Activation of an alternative meaning increases the difficulty of semantic processing that is widely equated with the increase of neuronal activity (e.g., [Federmeier, Kutas, 2001; Soldan et al., 2010; Gotts et al., 2012]).

Our results provide the evidence of common ERP correlates of semantic processing of ambiguous figures and verbal texts that were not considered in previous studies. These new results suggest that cognitive processes that are usually associated with semantic processing are also involved in the perception of ambiguous non-verbal information, such as the bi-stable pictures used here. However, detection of a semantic mismatch prior to perceptual reversion remains a missing link in this process that will need to be addressed in future studies.

CONCLUSION

In the present study, we attempted to look for cross-modal ERP correlates of ambiguity perception using non-verbal and verbal ambiguous stimuli. We found a common correlate of processing ambiguity contained in figures and verbal jokes. This correlate is an increase of N400 in response to ambiguous figures or verbal jokes, compared to presentation of non-ambiguous control figures and non-humoristic verbal stories. We tend to interpret this in a way that semantic processing is involved in both verbal jokes and ambiguous figures perception, in contrast to previous studies that did not at all consider semantic processing in assessing ambiguous figure perception and focused on early perceptual processes instead. The N400 increase points at an increased difficulty of semantic processing and implies that activation of both meanings requires switching to a higher cognitive load mode (that requires attention and top-down control), manifested as an increased neuronal activity. These findings need to be validated and extended in future studies which could employ a larger variety of ambiguous stimuli presented within different modalities (caricatures, comics, verbal riddles, proverbs, fables, etc) and use higher-resolution neuroimaging tools to specify the underlying activity sources neuroanatomically.

Список литературы

  1. Amsel B.D., Cree G.S. Semantic richness, concreteness, and object domain: An electrophysiological study. Canadian Journal of Experimental Psychology. 2013. 67: 117–129.

  2. Attardo S., Raskin V. Script theory revis (it) ed: Joke similarity and joke representation model. Humor: International Journal of Humor Research. 1991. 4 (3–4): 293–348.

  3. Boring E.G. A new ambiguous figure. American Journal of Psychology. 1930. 42: 444–445.

  4. Botvinick M.M., Cohen J.D., Carter C.S. Conflict monitoring and anterior cingulate cortex: an update. Trends in Cognitive Sciences. 2004. 8 (12): 539–546.

  5. Coulson S., Federmeier K.D., Van Petten C., Kutas M. Right hemisphere sensitivity to word-and sentence-level context: evidence from event-related brain potentials. Journal of Experimental Psychology: Learning, Memory, and Cognition. 2005. 31 (1): 129–147.

  6. Du Y., Zhang Q., Zhang J. X., Aronoff M. Does N200 Reflect Semantic Processing? – An ERP Study on Chinese Visual Word Recognition. PLoS ONE. 2014. 9 (3): e90794.

  7. Elkind D. Ambiguous Pictures for Study of Perceptual Development and Learning. Child Development. 1964. 35: 1391–1396.

  8. Enriquez-Geppert S., Konrad C., Pantev C., Huster R.J. Conflict and inhibition differentially affect the N200/P300 complex in a combined go/nogo and stop-signal task. Neuroimage. 2010. 51 (2): 877–887.

  9. Federmeier K.D., Kutas M. Meaning and modality: Influences of context, semantic memory organization, and perceptual predictability on picture processing. Journal of Experimental Psychology: Learning Memory and Cognition. 2001. 27 (1): 202–224.

  10. Feng Y.J., Chan Y.C., Chen H.C. Specialization of neural mechanisms underlying the three-stage model in humor processing: An ERP study. Journal of Neurolinguistics. 2014. 32: 59–70.

  11. Friedrich M., Friederici A.D. Early N400 development and later language acquisition. Psychophysiology. 2006. 43 (1): 1–12.

  12. Frings C., Groh-Bordin C. Electrophysiological correlates of visual identity negative priming. Brain Research. 2007. 1176: 82–91.

  13. Gajewski P.D., Falkenstein M. Effects of task complexity on ERP components in Go/Nogo tasks. International Journal of Psychophysiology. 2013. 87 (3): 273–278.

  14. Gao C., Hermiller M.S., Voss J.L., Guo C. Basic perceptual changes that alter meaning and neural correlates of recognition memory. Frontiers in Human Neuroscience. 2015. 9: 49.

  15. Gibson L., Atchley R.A., Voyer D., Diener U.S., Gregersen S. Detection of sarcastic speech: The role of the right hemisphere in ambiguity resolution. Laterality: Asymmetries of Body, Brain and Cognition. 2015. 21 (4–6): 549–567.

  16. Gotts S.J., Chow C.C., Martin A. Repetition priming and repetition suppression: Multiple mechanisms in need of testing. Cognitive Neuroscience. 2012. 3 (3–4): 250–259.

  17. Haro J., Demestre J., Boada R., Ferré P. ERP and behavioral effects of semantic ambiguity in a lexical decision task. Journal of Neurolinguistics. 2017. 44: 190–202.

  18. Hoffman P., Lambon Ralph M.A., Rogers T.T. Semantic diversity: A measure of contextual variation in word meaning based on latent semantic analysis. Behavior Research Methods. 2013. 45: 718–730.

  19. Huster R.J., Enriquez-Geppert S., Lavallee C.F., Falkenstein M., Herrmann C.S. Electroencephalography of response inhibition tasks: functional networks and cognitive contributions. International Journal of Psychophysiology. 2013. 87 (3): 217–233.

  20. Iannaccone R., Hauser T.U., Staempfli P., Walitza S., Brandeis D., Brem S. Conflict monitoring and error processing: new insights from simultaneous EEG–fMRI. Neuroimage. 2015. 105: 395–407.

  21. Intaité M., Koivisto M., Castelo-Branco M. Event-related potential responses to perceptual reversals are modulated by working memory load. Neuropsychologia. 2014. 56: 428–438.

  22. Intaite M., Koivisto M., Revonsuo A. Perceptual reversals of Necker stimuli during intermittent presentation with limited attentional resources. Psychophysiology. 2013. 50: 82–96.

  23. Kopp B., Rist F., Mattler U.W.E. N200 in the flanker task as a neurobehavioral tool for investigating executive control. Psychophysiology. 1996. 33 (3): 282–294.

  24. Kornmeier J., Bach M. EEG correlates of perceptual reversals in Boring’s ambiguous old/young woman stimulus. Perception. 2014. 43: 950–962.

  25. Kirsten M., Tiemann S., Seibold V.C., Hertrich I., Beck S., Rolke B. When the polar bear encounters many polar bears: event-related potential context effects evoked by uniqueness failure. Language, Cognition and Neuroscience. 2014. 29 (9): 1147–1162.

  26. Kubota J.T., Ito T.A. Multiple cues in social perception: The time course of processing race and facial expression. Journal of Experimental Social Psychology. 2007. 43 (5): 738–752.

  27. Kutas M., Federmeier K.D. Electrophysiology reveals semantic memory use in language comprehension. Trends in Cognitive Sciences. 2000. 4 (12). 463–470.

  28. Kutas M., Hillyard S.A. Reading senseless sentences: Brain potentials reflect semantic incongruity. Science. 1980. 207 (4427): 203–205.

  29. Lau E.F., Holcomb P.J., Kuperberg G.R. Dissociating N400 effects of prediction from association in single-word contexts. Journal of Cognitive Neuroscience. 2013. 25 (3): 484–502.

  30. Lei W., Zheng J., Lu Q. Event-related potentials and the decision making under risk and ambiguity. Multisensor Fusion and Information Integration for Intelligent Systems (MFI), 2014 International Conference on. IEEE, 2014.

  31. Martin R.A. The psychology of humor: An integrative approach. Academic press. 2010.

  32. Mayerhofer B. Neuro-Cognitive Processing of Garden Path Jokes: Theoretical Concepts and Empirical Correlates. 2014.

  33. Mayerhofer B., Schacht A. From incoherence to mirth: neuro-cognitive processing of garden-path jokes. Frontiers in Psychology. 2015. 6: 550.

  34. Morozova V.A., Yevtushenko S.K., Morozova T.M. Multifocal Event-Related Evoked Potentials in Early Diagnostics of Cognitive Disintegration: Clinical And Neurophysiologic Interpretation. Mezhdunar Nevrol Zhurn. 2012. 3: 26–41.

  35. Nieuwenhuis S., Yeung N., Van Den Wildenberg W., Ridderinkhof K.R. Electrophysiological correlates of anterior cingulate function in a go/no-go task: effects of response conflict and trial type frequency. Cognitive, Affective, & Behavioral Neuroscience. 2003. 3 (1): 17–26.

  36. Ortells J.J., Kiefer M., Castillo A., Megías M., Morillas A. The semantic origin of unconscious priming: Behavioral and event-related potential evidence during category congruency priming from strongly and weakly related masked words. Cognition. 2015. 146: 143–157.

  37. Özyürek A. Hearing and seeing meaning in speech and gesture: Insights from brain and behaviour. In: Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences. 2014. 369 (1651): 20130296–20130296.

  38. Rabovsky M., Sommer W., AbdelRahman R.A. The time course of semantic richness effects in visual word recognition. Frontiers in Human Neuroscience. 2012. 6, 11.

  39. Renn R.P., Cote K.A. Performance monitoring following total sleep deprivation: effects of task type and error rate. International Journal of Psychophysiology. 2013. 88 (1): 64–73.

  40. Sanguinetti J.L., Allen J.J., Peterson M.A. The Ground Side of an Object: Perceived as Shapeless yet Processed for Semantics. Psychological Science. 2014. 25 (1): 256–264.

  41. Siemann J., Herrmann M., Galashan D. fMRI-constrained source analysis reveals early top-down modulations of interference processing using a flanker task. Neuroimage. 2016. 136: 45–56.

  42. Soldan A., Habeck C., Gazes Y., Stern Y. Neural mechanisms of repetition priming of familiar and globally unfamiliar visual objects. Brain Research. 2010. 1343: 122–134.

  43. Van Veen V., Carter C.S. The anterior cingulate as a conflict monitor: fMRI and ERP studies. Physiology & Behavior. 2002. 77 (4): 477–482.

  44. Vergara-Martínez M., Comesaña M., Perea M. The ERP signature of the contextual diversity effect in visual word recognition. Cognitive, Affective, & Behavioral Neuroscience. 2017. 17 (3): 461–474.

  45. Vo M., Lauer T., Cornelissen T. Semantic integration without semantics? Meaningless synthesized scenes elicit N400 responses to semantically inconsistent objects. Perception. 2016. 45: 177–178.

  46. Wieser S., Wieser H.G. Event-related brain potentials in memory: correlates of episodic, semantic and implicit memory. Clinical Neurophysiology. 2003. 114 (6): 1144–1152.

  47. Wyer R.S., Collins J.E. A theory of humor elicitation. Psychological Review. 1992. 99 (4): 663–688.

Дополнительные материалы отсутствуют.