The Model Of Intermodal Convergence

Martin & Zappavigna (2019: 27-8):

As we stressed at the beginning of the paper building models of intermodality is facilitated if the descriptions of distinct modalities are informed by the same theoretical principles; and this is important for applications. Work in educational linguistics, for example Hood (2011) and Hao and Hood (in press), regularly has to deal with the interaction of language, paralanguage and imaging on Power Point slides. And for forensic linguistics, for example Martin and Zappavigna (2013) and Martin and Zappavigna, 2018, Zappavigna and Martin (2018), language and paralanguage interact with the semiotics of the location of the legal proceedings (which are very different for courtrooms and Youth Justice Conferences). The model of intermodal convergence (ideational concurrence, interpersonal resonance and textual synchronicity) presented in Table 2 above is far easier to operationalise when each of the modalities involved is interpreted from the perspective of SFL.

Blogger Comments:

[1] To be clear, the authors have concluded that paralanguage is an expression system of language.  That is, in their own terms, they have not provided a model of intermodality any more than proposing a phonological or graphological system of language would be a model of intermodality.

[2] To be clear, since the authors have concluded that paralanguage is an expression system of language, language and paralanguage do not interact, any more than language and phonology interact.

[3] To be clear, the model of intermodal convergence is one idea, redundantly given different names for each metafunction.  Moreover, the one idea is merely the superficial observation that different semiotic modes can make the same meaning.

More importantly, since the authors have concluded that paralanguage is an expression system of language, this model of intermodal convergence no longer applies, which, in turn, undermines the entire argument of the paper, given that the paper is predicated on the intermodal convergence of language and paralanguage (sonovergent vs semovergent).

On This Paper Clarifying The Theoretical And Descriptive Challenges Posed In Martin (2011)

Martin & Zappavigna (2019: 27):

This of course makes research into the relation between language and paralanguage an interesting case study as far as research into intermodality in general is concerned, possibly helping to clarify some of the theoretical and descriptive challenges posed in Martin 2011.

Blogger Comments:

[1] To be clear, if, as the authors argue, paralanguage is an alternative expression plane of language, then the relation between language and paralanguage is not 'intermodal', since language and paralanguage are just two perspectives on the same semiotic mode.

[2] To be clear, given the wealth of theoretical confusions in this paper that have been identified here, any clarifications of any theoretical and descriptive challenges are purely accidental.

On the other hand, ignoring the fact that Martin (2011) begins by misunderstanding Saussure's sign (pp243-5), and the relation between Saussure's sign and linguistics (p245), and ignoring all the other theoretical misunderstandings that follow, the questions posed by Martin can be listed here so that the reader can assess which of them this paper has helped to clarify.

p245:

Based on this reading of Saussure one could ask of any multimodal analyst:

1. Do you conceive of the sign as an entity that realises a meaning located outside itself (in the material world or in the mind or elsewhere) or alternatively as a meaning construing act?
2. Where and how, if at all, do you explicitly model valeur (i.e. the system of differences among signs)?

pp246-7:

Based on this reading of Hjelmslev and Halliday, one could ask of any multimodal analyst:

1. For a given semiotic system, how many strata are you proposing, and on which stratum is your description located?
2. Are your strata related by metaredundancy (as patterns of patterns)?
3. Are there distinct systems of valeur on each of the strata you propose?
4. Is there any ontogenetic or phylogenetic evidence suggesting that any stratified system you propose evolved from an unstratified or a less stratified system?

p247:

Based on this reading of Halliday, one could ask of any multimodal analyst:

1. For a given stratum, how many ranks are you proposing, and at which rank is your description located?
2. Are there distinct systems of valeur on each of the ranks you propose?
3. Are your distinct systems of valeur related by constituency (as parts to wholes)?

p248:

Based on this reading of Halliday, one could ask of any multimodal analyst:

1. For a given semiotic system, how many metafunctions are you proposing?
2. Are there topologically distinct systems of valeur for each of the metafunctions you propose?
3. By what criteria are systems of valeur seen as relatively independent or interdependent of one another?

p249:

Based on this reading of Halliday one could ask of any multimodal analyst:

1. For a given semiotic system, how many kinds of structural realisation are you proposing?
2. Are the different types of realisation associated with different types of meaning?
3. When analogising from metafunctions in language to your semiotic system did you take kinds of meaning or types of structure as point of departure?

p250:

Based on this reading of paradigmatic and syntagmatic relations in SFL one could ask of any multimodal analyst:

1. Are your descriptions formalised as system/structure cycles, explicitly showing the relation of systemic choices to structural consequences?
2. How many system/structure cycles are you proposing and how are they related to one another (by strata, rank, metafunction or some other theoretical parameter)?

p252:

Based on this reading of system and text in SFL one could ask of any multimodal analyst:

1. Is the complementarity of realisation and instantiation addressed your description?
2. If so, how are you distinguishing axial realisation (the defining interdependency of system and structure) from instantiation (the logogenetic unfolding of realisational resources as text)?
3. As far as the contextual specification of your system is concerned, what genres/registers/text types do you propose?

p254:

In relation to Matthiessen’s proposed cline of integration one could ask the multimodal analyst:

1. Do you manage intermodality by proposing a single system of valeur, on a higher stratum or not, realised axially or inter-stratally by two or more modalities (realisational integration); or do you propose a coupling process weaving together meanings from different modalities in a single text (instantial integration)?

p255:

Based on these intermodal integration and complementarity issues one can ask the multimodal analyst:

1. Are the relations you recognise as obtaining between modalities in an intermodal text the same as those you find between units of a text in a monomodal one?
2. Do you recognise different kinds of intermodal relations depending on the kind of meaning involved (ideational/interpersonal/textual)?

p256:

Based on this discussion of affordances and commitment one could ask the multimodal analyst:

1. How do you model the amount of meaning committed and thereby the complementary contribution of different semiotic systems in an intermodal text?
2. How does the semantic weight of a given system’s contribution reflect its affordances?

p260:

In light of this reading of Cléirigh, one could ask:

1. Is the semiotic system you are working on a denotative semiotic system, with its own content form and expression form?
2. If not, does it involve parametric resources of the kind outlined by van Leeuwen (i.e. multiple, simultaneous, graded, binary systems)?
3. If so, could it be usefully factored into protosemiotic, denotative semiotic and epilinguistic systems?

p262:

In light of these concerns with identity and affiliation, one could ask:

1. How do you describe the allocation of the semiotic resources you are focusing on to repertoires of users?
2. How do these repertoires engender communities of such users?
3. Is there a distinctive role for denotative semiotic, protosemiotic and episemiotic systems in this process?

pp263-4:

In light of these concerns with the limits of semiosis, one could ask:

1. On what basis do you distinguish between the semiosis you are considering and its biological and/or physical environment?
2. To what extent do you feel that interdisciplinary research involving neurobiologists and/or physicists is necessary to give a full account of the discourse you are considering?
3. Are you deliberately treating aspects of biological and physical materiality as if they were semiosis?

What The Authors Have Done In This Paper

Martin & Zappavigna (2019: 25):

In this paper we have outlined a model distinguishing behaviour from meaning (somasis vs semiosis), and within semiosis, language from paralanguage. Paralanguage itself was then divided into sonovergent and semovergent systems according to their convergence with either the expression plane or content plane of language.

Blogger Comments:

[1] To be clear, the model the authors have outlined is (part of) Cléirigh's model, linguistic and epilinguistic body language, though misunderstood and rebranded as their own systems, sonovergent and semovergent paralanguage.

[2] As previously demonstrated here, having distinguished non-semiotic behaviour from semiosis, the authors then interpret non-semiotic behaviour as semiotic.

[3] As previously demonstrated, the paralanguage that the authors rebrand as 'sonovergent' is actually, in their own terms, 'semovergent', not 'sonovergent', because it instantiates the same meanings as language, but it diverges from language in the way it is expressed, gesturally rather than vocally.

On the other hand, the authors' approach to semovergent paralanguage has been merely an unsuccessful attempt to fit gestural data to Martin's discourse semantic systems, instead of using the gestural data to encode theory.  This will lead them to the erroneous conclusion (p26, 28) that paralanguage is an alternative expression form of language, alongside phonology, graphology and sign.

Mime As Paralanguage

Martin & Zappavigna (2019: 22-6):

In terms of the TONALITY of this sequence, there are two miming segments where tone groups might have been. For each, the vlogger mimes the paralanguage of her parking spot assailant. In the first slot she mimes his interpersonal attitude paralanguage, as discussed in section “Evaluation (interpersonal semovergent paralanguage)” above (Fig. 39).

 

In the second she mimes his ideational motion paralanguage as she twice gestures  leaving (the second time including a textual pointing gesture) (Figs. 40 and 41). 

The third time his motion gesture is mimed in fact concurs with language (Fig. 42). 

As we can see, the two miming segments are heavily co-textualised by language that makes explicit what is going on. The orientation to the narrative introduces the recurrent problem of someone following the vlogger in a parking lot and waiting for her to leave. The miming segments are themselves introduced with the incomplete tone group // he was like... //, with a missing tonic segment. The vlogger then mimes the expected information, before making it linguistically explicit in a tone group converging with the third iteration of the gesture.

Setting aside pantomime (the ‘art of silence’ Marcel Marceau referred to it), we can predict that co-textualisation of this kind is a generalisable pattern as far as semovergent paralanguage in the absence of language is concerned. What the moment of mime does not provide as far as language is concerned, the immediately preceding and following co-text does provide. So the convergent nature of semovergent paralanguage is clear.

Blogger Comments:

[1] This is misleading, and motivated by the authors' desire to match data to theory — their claim that the tone group provides an essential unit of analysis for work on paralanguage — rather than to use data to encode theory, and theory to decode data.  To be clear, there are two miming segments where language might have been.  That is, the authors could just as easily have said there are two miming segments where clauses might have been, but this would not have matched their hypothesis.

[2] To be clear, the authors have previously (p21) defined mime as semiosis that does not accompany language:

In terms of our model mime is semovergent paralanguage that does not accompany language, an apparent contradiction in terms.

Nevertheless, here they present mime as semiosis that does accompany language.

[3] To be clear, the vlogger mimes the body language of her non-speaking "parking spot assailant".  In doing so, on Cléirigh's original model, she deploys epilinguistic body language ("semovergent paralanguage") to mime the protolinguistic body language of the frustrated motorist.  This is analogous to deploying pictorial epilanguage to represent protolanguage.  That is, contrary to the authors' claim, the body language of the motorist is neither paralanguage nor semovergent.

[4] To be clear, on Cléirigh's original model, the vlogger, through her arm gestures, deploys epilinguistic body language ("semovergent paralanguage") in miming the epilinguistic body language of the frustrated motorist. And simultaneously, through her facial expressions, she again deploys epilinguistic body language ("semovergent paralanguage") in miming the protolinguistic body language of the frustrated motorist.

[5] To be clear, unknown to the authors, this instance of language presents the meaning of the body language that immediately follows as the focus of New information.

[6] "Oh no it isn't!"  It was mime that Marcel Marceau referred to as the "art of silence", not pantomime. (Ordinarily, contrary to Kendon's use of the term, 'pantomime' refers to a type of musical comedy staged for family entertainment, especially around Christmas / New Year.)

[7] To be clear, here the authors claim that the "co-textualisation" of mime by language is what happens in the absence of language.

[8] Here the authors argue that the meanings not provided by mime "converge" with the meanings provided by language.  That is, the absence of meaning "converges" with meaning.

The Claim That Units Of Speech Rhythm Realise Elements Of Writing Pedagogy

Martin & Zappavigna (2019: 19-20, 22-3):

As far as longer wave lengths of information flow are concerned,²³ our vlogger is seated and so whole body movement from one location to another is not a factor (as it would be for example for a lecturer roaming to and fro across a stage; cf. Hood 2011).

²³ van Leeuwen (1985, 1992) and Martinec (2002) argue that SFL’s phonological hierarchy can be pushed up several wave lengths beyond the tone group and their work suggests that higher level rhythm would converge with higher level periodicity in Martin's framework.

Blogger Comments:

To be clear, the claim here is that proposed higher level phonological units "converge" with Martin's discourse semantic functions of macro-Theme, hyper-Theme, hyper-New and macro-New.  There are several obvious theoretical inconsistencies here.

The over-arching inconsistency is that the authors are proposing that patterns of speech rhythm correspond to pedagogical suggestions on how to write.  This is because Martin's four discourse semantic functions are actually rebrandings of introductory paragraph, topic sentence, paragraph summary and text summary, as previously explained.

A second inconsistency is that speech rhythm can only identify potential New information, and bears no systematic relation to thematicity.

A third inconsistency is that the use of gesture to realise New information is linguistic body language ("sonovergent" paralanguage), not epilinguistic body language ("semovergent" paralanguage).

A fourth inconsistency is the matching of structural units (wavelengths beyond the tone group) with elements of structure (Themes and News).

A fifth inconsistency, in the authors' own terms, is the use of their term for a relation between the same stratum of different semiotic systems, converge, for an interstratal relation within language.

The Semovergent Paralanguage Of ENGAGEMENT

Martin & Zappavigna (2019: 18, 20, 21):

Turning to ENGAGEMENT, Hood notes the significance of hand position as far as supporting the expansion and contraction of heteroglossia is concerned – with supine hands opening up dialogism and prone hands closing it down. In the following example the vlogger’s supine hands converge with the modalisation probably, reinforcing acknowledgement of the viewers voice (Fig. 31).

 

Two moves later the hands flip over to prone position in support of the negative move shutting down the expectation that the vlogger was in control of the new colour of her hair (Fig. 32).

Blogger Comments:

[1] This is misleading; the speaker's handshape does not "converge" with modalisation probably. To be clear, the speaker's handshape is timed with the tonic hair.  The timing of the gesture thus instantiates textual linguistic body language ("sonovergent" paralanguage), highlighting hair as the focus of New information.  On this basis, the handshape instantiates ideational epilinguistic body language ("semovergent" paralanguage), realising hair.

In this first instance, the authors have again tried to make the data fit their theory, instead of using the data as a resource for theorising.

[2] This is misleading; the speaker's hands are not in a prone position — lying flat, palm downwards — in this instance.  Instead, each hand has the tips of the thumb and curved forefinger touching to form a horizontal circle, with the other fingers below them and similarly curved.  This handshape is consistent with holding an object, such as a bottle of hair dye, which would be an instance of ideational epilinguistic body language ("semovergent" paralanguage).

In this second instance, the authors have again tried to make the data fit their theory, instead of using the data as a resource for theorising.

As in the first instance above, the gestures also realise the meanings of linguistic body language ("sonovergent" paralanguage).  In terms of the textual metafunction, both hands beat down on the salient syllables not and find, highlighting both Finite and Predicator, and then on the tonic hair, marking  the Complement hair dye as the focus of New information.  In terms of the interpersonal metafunction, both hands stay level for the tonic segment (hair dye that I), in line with the level/low-rising tone choice (tone 3).  (Note that this tone group is incorrectly analysed as tone 4 by Smith, which, with declarative MOOD, would realise the KEY meaning 'reservation'.)

Lastly, the reader may also want to consider why the speaker would need to shut down the possibility of other points of view on the proposition I could not find the hair dye that I bought previously when I dyed my hair.

Paralanguage "Resonating With" GRADUATION: FOCUS

Martin & Zappavigna (2019: 17, 19):

Alongside paralanguage of this kind converging with FORCE, Hood notes the potential for precise hand shapes and muscle tension to resonate with FOCUS. In the following example, introduced in section “Representation (ideational semovergent paralanguage)” above, the vlogger tightens her grip on the tiny virtual needle she is holding and frowns slightly in concentration as she role plays the precision involved in the dermatologist piercing her bumps (Fig. 29).

Blogger Comments:

To be clear, FOCUS is a system of GRADUATION, which is concerned with the intensification of ATTITUDE in the interpersonal system of APPRAISAL.  Here, as the instance of language makes clear, no interpersonal assessment is being enacted, in terms of AFFECT, APPRECIATION, or JUDGEMENT, and so there is nothing to graduate in terms of FOCUS.

Moreover, in this instance, the authors have simply confused the focus of attention ('concentration') of the speaker with FOCUS as a system of APPRAISAL.

Paralanguage "Converging With" GRADUATION: FORCE

Martin & Zappavigna (2019: 17, 19):

The most striking example of intensification in the hair colour episode occurs when the vlogger uses whole body movement to enact her reaction to how dark her hair is. She throws her head back and leans back as her arms move rise up – literally overwhelmed with emotion (Fig. 28).

Blogger Comment:

[1] To be clear, it is not that the whole body expresses the same meaning in this instance, but that the authors have not analysed the different meanings being made by the various gestures and postures, including the shift of gaze.

[2] To be clear, the intensification in this instance is of the Quality dark, which is ideational in function, and quite distinct from the speaker's hatred of the Quality, which is construed by the following clause.  That is, the intensification is a feature of the assessed, not of the assessing (e.g. I really hate it).   This is demonstrated by the fact that the arm gesture beats on the tonic so, the intensifier of dark.

In terms of Cléirigh's original model, the beating of the gesture on the tonic is linguistic body language ("sonovergent" paralanguage), highlighting so as the focus of contrastively New information, whereas any aspects of the body language expressing conscious states are instantiations of paralinguistic body language.  That is, contrary to the authors' claim, no aspects of this instance of body language can be identified as epilinguistic ("semovergent").

[3] The claim that this gestural configuration expresses 'being overwhelmed by the emotion of hate' — literally or figuratively — requires considerable justification, none of which is given.

[4] To be clear, Figure 28 displays an (incomplete and) incorrect phonological analysis — the tonic  actually falls on so, not dark, the initial foot is omitted, and the pronoun I begins the following tone group (after a silent Ictus):

//1+ and it's / so dark //

Misrepresenting Cléirigh's Work As The Authors' Work

Martin & Zappavigna (2019: 8, 6):

Sonovergent paralanguage converges with the prosodic phonology of spoken language (Halliday 1967, 1970; Halliday and Greaves 2008; Smith and Greaves 2015). From an interpersonal perspective, it resonates with tone and involves a body part (e.g. eyebrows or arms) moving up and down in tune with pitch movement in a tone group (TONE and marked salience). From a textual perspective it involves a body part (e.g. hands, head) beating in sync with the periodicity of speech – which might involve beats aligned with a salient syllable of a foot, the tonic syllable of a tone group, or a gesture co-extensive with a tone group (i.e. in sync with TONALITY, TONICITY or RHYTHM). An outline of this sonovergent paralanguage is presented in Table 4.

 

Blogger Comments:

[1] To be clear, as previously explained, 'sonovergent' paralanguage is the authors' rebranding of Cléirigh's 'linguistic' body language, justified on the basis that the invented word 'sonovergent' is more transparent.  As previously explained, linguistic body language, as the name implies, is "convergent" with language itself and differs from language only in its mode of expression — the opposite of the authors' claim.  As Cléirigh originally elaborated:

	lexicogrammar	prosodic expression
		phonology	kinetic
textual	LEXICAL SALIENCE	RHYTHM	gesture (hand, head) in sync with the speech rhythm
	FOCUS OF NEW INFORMATION	TONICITY	gesture (hand, head) in sync with the tonic placement
	INFORMATION DISTRIBUTION	TONALITY	gesture (hand, head) co-extensive with tone group
interpersonal	KEY	TONE	gesture (eyebrow*, hand) in tune with the tone choice

* also: rolling of the eyes for tone 5.

[2] This is misleading.  Here the authors elaborate the details of Cléirigh's model of linguistic body language — see [1] — as if it is their own development of it as sonovergent paralanguage.  Proof that the omission of attribution is not accidental is provided by the misleading claim (p3) identified earlier which primes the reader for the interpretation of this work as the author's innovation:

We will in fact suggest that SFL’s tone group, analysed for rhythm and tone, provides an essential unit of analysis for work on paralanguage as far as questions of synchronicity across modalities are concerned.

The Pretext For Rebranding Cléirigh's Body Language Systems As The Authors' Paralanguage Systems

Martin & Zappavigna (2019: 6, [5], 7, [6]):

In their work on intermodal relations in children’s picture books (Painter and Martin 2012; Painter et al. 2013) Painter and her colleagues suggest a model involving degrees of convergence between verbiage and image. The model is organised by metafunction – degrees of concurrence for ideational meaning, degrees of resonance for interpersonal meaning and degrees of synchronicity for textual meaning (for illustrative text analysis see Martin 2008, Painter and Martin 2012). The relevant terminology is presented in Table 2 below.

We have drawn on this terminology to deal with two dimensions of the relation between language and paralanguage introduced by Cléirigh as ‘linguistic body language’ and ‘epilinguistic body language’. The basic distinction here is between paralanguage that is in tune with (resonance) or in sync with (synchronicity) the prosodic phonology (i.e. rhythm and intonation) of spoken language on the one hand and on the other paralanguage that expresses meanings made possible by having language – in Cléirigh’s terms linguistic vs epilinguistic body language respectively. We have preferred a more transparent terminology, derived from Table 2, with phonologically convergent paralanguage referred to as sonovergent and semantically convergent paralanguage as semovergent. This revised terminology is outlined in Table 3.

 

Blogger Comments:

[1] See the earlier clarifying critique The Notion Of Intermodal Convergence.

[2] To be clear, as previously explained, this model of intermodal relations merely compares the semogenesis of different semiotic modes and labels any similarity as 'convergence'.  So, in modelling the relations between language and paralanguage, it is limited to simply comparing them and noting any similarity; see [4] below.

[3] To be clear, the authors' pretext for rebranding Cléirigh's systems is that their newly invented words, 'sonovergent' and 'semovergent', are more "transparent"; see [4] below.

[4] To be clear, here the authors confuse a purported relation between semiotic modes — between language and paralanguage — with semiotic modes themselves, and rebrand the semiotic modes, linguistic and epilinguistic body language, in terms of the purported relation between them and language.

In considering the purported relations between language and paralanguage, as previously explained, the 'convergence' model is limited to the notion of similarity, so the authors are limited to identifying some point of similarity between language and paralanguage.  This leads them to propose that:

• linguistic body language is phonologically convergent with language ('sonovergent'), and
• epilinguistic body language is semantically convergent with language ('semovergent').

These claims can be examined in turn.

The problem with the claim that linguistic body language is phonologically convergent with language is that it is the exact opposite of what is true.  Being 'linguistic', linguistic body language is "convergent" with language itself.  Where it differs from language is in its mode of expression, visible body movements, usually silent, instead of audible sounds of the vocal tract.  This can be made more explicit by looking at the source of the authors' ideas, Cléirigh's model:

	lexicogrammar	prosodic expression
		phonology	kinetic
textual	LEXICAL SALIENCE	RHYTHM	gesture (hand, head) in sync with the speech rhythm
	FOCUS OF NEW INFORMATION	TONICITY	gesture (hand, head) in sync with the tonic placement
	INFORMATION DISTRIBUTION	TONALITY	gesture (hand, head) co-extensive with tone group
interpersonal	KEY	TONE	gesture (eyebrow, hand) in tune with the tone choice

By the authors' logic, graphology might also be described as "phonologically convergent" with language.

The problem with the claim that epilinguistic body language is "semantically convergent" with language is that it risks confusing bi-stratal epilanguage with tri-stratal language — a point that will be taken up in future posts that examine the authors' analyses of data.  A further problem is that epilinguistic body language is not exclusively paralinguistic — it can be used in the absence of language, as the practice of mime demonstrates.