Collaborative problem solving (CPS) is a 21st-century skill essential for learning gains, workplace success, and tackling increasingly complicated global problems. Group diversity plays a vital role during collaborative activities, especially in a digital space. Although CPS involves dynamic communication behaviors, few studies have considered the impact of cultural diversity on the complex and reoccurring discourse involved in CPS tasks. In this study, we explore team conversations during a CPS task to understand the role of cultural diversity on team communication patterns. First, we characterized team dialogues with an existing CPS framework; then used recurrence quantification analysis (RQA) to quantify group communication and capture recurrent patterns. Finally, we compared the patterns across groups with varying degrees of cultural diversity. Our results suggest that groups with higher levels of cultural diversity, compared to more homogeneous groups, had a higher number of group messages, spent more time in group discussions, and demonstrated greater convergence and complexity in communication patterns. These intricate and complicated communication patterns support the notion that cultural diversity can produce both positive and negative outcomes and may explain the perception of cultural diversity in teams as a “double-edged sword”.
Collaborative problem solving (CPS) involves individuals working together to solve a problem and promotes division of labor, sourcing information from different perspectives and backgrounds, and increasing innovation and creative solutions that stem from the presence of multiple group members . CPS has been identified as a key 21st century skill [9, 80] that plays a pivotal role in both workplace  and educational environments . Furthermore, CPS has increasing international importance and has been shown to be an essential skill needed across several domains to solve complex environmental, social, and public health problems [52, 29]. From a socioeconomic perspective, the rise of complicated global issues requires innovative solutions derived from individuals working together and generating solutions from diverse perspectives . In professional environments, employers often report problem-solving [49, 60] and collaboration  as skills that are essential for the success and employability of recent college graduates. In educational contexts, CPS is also a key component for successful teamwork and student learning [52, 27, 28]. Positive collaborative interactions have been shown to improve student psychological and performance outcomes [1, 35, 34, 44, 42, 41, 18, 20, 62].
Given the pivotal role of collaboration across multiple disciplines, it is unsurprising that an increasing number of studies have explored which team attributes are important for successful CPS outcomes. For example, studies have considered the impact of group size [68, 45, 57], group diversity [33, 66, 6, 21, 7, 75], and personality differences [37, 32] on team performance. This area of research has consistently highlighted that team diversity in general, and cultural diversity in particular, play a critical role in successful team collaboration [66, 7, 75, 33]. Notably, this line of previous work has generally focused on measuring the effects of team diversity on static post-collaboration measures (e.g., performance outcome) [66, 33, 77]. However, there have been limited research efforts devoted towards understanding the role of diversity from a more dynamic, process-oriented perspective, which are fundamental to collaborative interactions [19, 16, 42]. As such, current studies on diversity in teams are limited in offering insight on many interdependent aspects within collaborative interactions such as negotiation, coordination, and regulation, among others. In order to explore how these fine-grained collaborative interactions are influenced by diversity, more nuanced techniques are needed.
To address this gap, we use a dynamical systems lens to explore cultural diversity in teams and team communication patterns [58, 15, 79]. Specifically, we apply recurrence quantification analysis (RQA)  to quantify team dynamics, and capture recurrent patterns of interaction amongst the members. Using this novel approach, we aim to uncover low-level temporal patterns in CPS communication and variations therein across groups with different levels of cultural diversity. Group communication is inherently interdependent and research has shown that group composition factors are associated with different aspects of social and cognitive processes during collaborative interactions [20, 19, 42, 55, 11, 10, 17]. Therefore, we are motivated to examine whether cultural diversity as a team composition factor is associated with different collaborative communication dynamics and structures.
The remainder of the paper is organized as follows. First, we review the literature on diversity in teams and the impact of diversity on team outcomes and communication. Second, we provide an overview of the current practices in quantifying communication in CPS tasks and the previous work related to RQA in the context of teams and group dynamics. Third, we present our methodological approach including a description of our CPS task, the qualitative coding of CPS skills exhibited in group communication, and the RQA measures. Finally, we present our RQA analysis results with regards to cultural diversity in groups as well as discuss our findings and their implications for understanding communication behaviors in CPS.
2. RELATED WORK
2.1 Diversity in Teams
Diversity in teamwork and its impact on group’s performance outcomes has been studied extensively across multiple disciplines and contexts (c.f. [33, 21, 72, 77, 66], for meta-analyses on this topic). However, studies have demonstrated mixed findings on the impacts of team diversity on various team outcomes. Horwitz & Horwitz, 2007 found a positive association between task-related diversity in groups and performance, yet no direct relationship between demographic diversity and group performance. There is some evidence to show that diversity in groups can result in positive outcomes, such as increased innovative and creative ideas [33, 66]. However, other studies suggest that diversity can result in negative outcomes such as increased conflict and lower group cohesion [65, 66]. Although the direct impact of diversity on team outcomes remains unclear [8, 66, 77], several studies have demonstrated that different sources of diversity (personality, cognition, gender, race, ethnicity, etc.) can impact groups to varying degrees [59, 33]. The impact of diversity is often associated with the type of diversity: surface-level diversity (i.e., easily observable attributes such as gender, race, ethnicity) v.s. deep-level diversity (i.e., less overt attributes such as personality, cognition, values/beliefs) [59, 31]. Initially, surface-level attributes may influence group dynamics, but over time deep-level attributes may become more salient and influential . Considering the interactions among the participants in this study and many other collaborations are short-term (20-minute discussions), surface-level diversity and cultural diversity in particular can prove to be highly influential on group interactions. In the current study, we focus on a surface-level attribute, cultural diversity operationalized by ethnic compositions in group, and its relationship to the communication dynamics during CPS.
2.2 Cultural Diversity in Teams: A Double
Although there are various forms of group diversity, cultural diversity has often been of particular interest to scientists because of the increased globalized and interconnected workforce [7, 71]. Moreover, an array of collaboration technologies and online platforms enable distance teamwork, and create greater collaborative opportunities for individuals with different cultural backgrounds[22, 29]. Previous studies suggest that the role of cultural diversity on team performance is nontrivial. However, the effect of cultural diversity on group performance is mixed. These effects are often mediated by factors such as creativity, conflicts, communication effectiveness, social integration, and satisfaction . For example, although culturally diverse teams benefit from higher levels of creativity and satisfaction, they also suffer from lower social integration and more conflicts . In addition, it is commonly assumed that people with different cultural and ethnic backgrounds are likely to hold different views and priorities when communicating with others [65, 14]. Following this assumption, higher degrees of cultural diversity in groups may contribute to more complex interpersonal dynamics and information sharing behavior. To extend our understanding of communication patterns associated with cultural diversity, we took on a dynamical system lens to examine structural and temporal processes in CPS discourse.
2.3 Quantifying the Impact of Cultural
The literature on cultural diversity in teams covers a wide range of domains and collaborative contexts. Existing studies have extensively focused on static team or individual outcomes such as performance [70, 76], number or quality of ideas generated [50, 53], decision-making tasks [81, 40, 64], achieved learning , and psychological measures such as learner experience and attitude towards group interactions . These studies have extended our knowledge on the relation between cultural diversity and the end product of the collaboration. However, few studies [69, 73] have considered how cultural diversity impacts more dynamic aspects of the group collaboration process, such as language and group discourse. For example, Tenzer et al., (2014) found that cultural-based language differences can impact group perceptions of trust and competencies. CPS is inherently interactive and dynamic. Collaborative interactions involve reoccurring and interrelated discourse as teams exchange ideas, negotiate, and share information. Thus, it is necessary to unpack collaborative process with regards to cultural diversity compositions. To address this gap, our study focuses on the fine-grained temporal communication patterns across cultural diverse and culturally similar groups.
2.4 Quantifying CPS Communication
Collaborative problem-solving involves dynamic interpersonal exchange and shared cognitive behavior of individuals [19, 12]. Successful CPS requires multiple skills and subskills for effective communication stages, including negotiation, information sharing, coordination and so on. Language is considered a less intrusive means compared to traditional survey sampling to reveal cognitive processes of the human mind. A number of existing assessment frameworks have been developed to identify and capture CPS skills from interactive dialogues. For example, the Programme for International Student Achievement (PISA) , identifies a framework with three social competencies (i.e., establishing and maintaining shared understanding, taking appropriate action to solve the problem, and establishing and maintaining team organization) and four cognitive processes (i.e. exploring and understanding, representing and formulating, planning and executing, and monitoring and reflecting). This framework was first introduced by OECD to evaluate CPS skills during various computer-simulated assessment tasks, highlighting important aspects unique to collaborative interactions mediated by computers. Other CPS frameworks that subsequently surfaced follow similar ontology. For instance, Liu et al. (2016)  proposed a framework that conceptualizes four broad CPS skills: sharing ideas, assimilating and accommodating knowledge/perspective taking, regulating problem solving ideas, and maintaining positive communication. Andrews-Todd & Forsyth mapped out more nuanced skills under the social and cognitive domains . More recently, some researchers propose to conceptualize CPS interactions as a continued sociocognitive spectrum, in contrast to the previous dichotomous view on social and cognitive processes . Through this lens, CPS practices could best be described as sociocognitive in nature, allowing more opportunities for adopting computational methods to meaningfully capture natural language patterns of learners. Increasingly, studies have suggested that natural language processing and other artificial intelligence techniques are effective ways to measure CPS skills [13, 74]. Given this trend, researchers have called for more efforts towards developing and adapting innovative data mining methods to effectively quantify CPS patterns and make sense of learner communication behavior .
2.5 Communication in Teams as a Dynamical System: RQA
Communication and CPS skills can be considered a dynamic and complex system [46, 23, 36]. Several methods can be used to quantify and analyze group dynamics. However, Knight, et al. (2016) point out that each are limited by a lack of consideration for time dependencies and therefore oversimplification of group dynamics . RQA, on the other hand, is a non-linear dynamical systems approach that enables the evaluation of recurring patterns across times . RQA has be widely used recently to study social interactions, group communication, and group dynamics [3, 4, 36, 25, 26, 15, 67]. For example, Fusaroli & Tylén (2016) took advantage of RQA and cross-recurrence quantification analysis (CRQA) to analyze dyadic conversations patterns and subsequent performance. Based on RQA and CRQA, they defined measures of interactive alignment, interpersonal synergy, and self-consistency. Interpersonal synergy serve as a significant predictor of performance suggesting the importance of complimentary conversations between the dyads. This dynamic system approach, to our knowledge, has not yet been applied to investigate the communication group dynamics in culturally diverse and homogeneous teams.
In the current study, we use RQA to measure structural components of groups’ conversations. Specifically, we focus on time-series data that map CPS skills exhibited in student discourse based on the ontology presented in Andrews-Todd et. al., . RQA results can be visualized using a recurrence plot. The recurrence plot gives a visual representation of the components of each time-series data and plots their recurrence over time. Figure 1 provides an example of RQA in the context of CPS skills. Figure 1A depicts chat discussions during a typical CPS task. Each color in the chat discussion depicts a coded CPS skill (e.g., information sharing) that results in a times series of skills used labeled in Figure 1B. Figure 1C illustrates a recurrence plot based on the chat discussions in Figure 1A. The CPS skills on X and Y axis correspond to the temporal sequence of chat utterances and each dot is an instance in which the CPS skill has recurred in the group conversation. The central diagonal line represents the sequence of CPS skills plotted against themselves, also known as the line of identity (LOI). The recurrence of a skill code used previously in the conversation is represented by dots in the recurrence plot. The recurrence points along with their patterns and alignment provide insight to structural components of group communication. For example, the recurrence of Sharing Information (green) took place at time points four, five, nine, and eleven and the recurrence of Shared Understanding (orange) on times three and eight are recurrence points.
Of particular interest, is any group of dots that create a diagonal line and that are not part of the LOI. A diagonal line of length is an indication of a recurring pattern of skill codes in the same sequence. For example, the sequence Monitoring (purple), Shared Understanding (orange), and Information Sharing (green) creates a diagonal line of length . These exhibited CPS skills occurs back to back at position, two, three, and four and then again at seven, eight, and nine, thus creating a recurrence of CPS skills.
3. CURRENT WORK
CPS tasks involve dynamic discourse and communication between groups. Therefore, by understanding how cultural diversity impacts these dynamic communication patterns and CPS skills, we can gain insight on the impact of cultural diversity beyond outcome measures such as CPS performance. The current study investigates how cultural diversity in teams impacts group communication behaviors from a dynamical system lens. We explore the linguistic CPS patterns that emerge in culturally homogeneous and diverse groups of undergraduate students as they complete an online CPS task. Specifically, we aim to explore the following research questions:
RQ1: Does cultural diversity impact structural components (as captured with RQA) of a group’s conversation?
RQ2: How does cultural diversity impact structural components (as captured with RQA) of a group’s conversation?
For RQ1, we examine if the degree of cultural diversity in teams is related to different CPS skills discourse patterns. Given that previous findings demonstrate that cultural diversity can impact team communication [69, 73], we hypothesize that we would detect differences in CPS skills across teams with varying degree of cultural diversity. For RQ2, we use RQA to explore how CPS skills’ patterns of communication are impacted by various degrees of cultural diversity ranging from cultural homogeneous to cultural heterogeneous groups.
Figure 2 outlines the methodological approach and research workflow that was employed in the current study. First, we sourced student data from an online CPS task completed in small groups with varying degrees of cultural diversity . Second, we extracted chat discourse data from these online interactions and characterized the discourse with a CPS skill framework . Third, RQA was applied to the coded data to generate recurrence plot and RQA measures for each conversation. Finally, Kruskal-Wallis test was used to compare
the structural components of group conversation (obtained from RQA) across groups with varying degrees of cultural diversity. In the next section, we provide further details of our methods to evaluate the dynamic communication patterns across culturally diverse groups.
A total of undergraduate students from a large university in the U.S. southwest participated in the study. Participants were randomly assigned into teams to take part in a Hidden Profile CPS task (described below). In total teams were included in the study. Teams are predominantly four person groups with a few exceptions that consist of three or five people. In this study we consider four-person groups only for the sake of consistency in analysis. Over a half of our participants were female (N = 347), and most of them were freshman () or junior (). The average age of the participants was 19.5 years, with the 90% of the participants between the ages 18 to 22 years old. Of those participants who reported their race and ethnicity (497 out of 514), 62 (12%) of the participants were White, 9 (1.7%) were Black or African American, 209 (40.6%) were Asian or Asian American, 162 (31.5%) were Hispanic or Latino, and 9 (1.7%) were multiracial. Out of 502 participants who reported their first language the breakdown was as follows: 142 English, 152 non-English, and 208 bilingual. Additionally, over half of students () identified as first-generation students.
Participants were randomly assigned into teams of four individuals to complete a decision-making task on the Education Platform for Collaborative Assessment and Learning (EPCAL) . EPCAL is a platform by Educational Testing Service that provides a collaboration space for participants to communicate, for teachers or organizers to manage the participants and team formations, and for researchers to study team collaboration in a computer-mediated environment. Prior to beginning the task, participants were asked to complete a background survey to collect information on race, gender, education level, and native language. Next, students were prompted with a problem (e.g., “choose the best apartment”) and were asked to rank three options based on positive features (e.g., “this apartment is at a prime location”) and negative features (e.g., “the rent is expensive”). Teams were randomly assigned to one of four decision-making scenarios including ranking apartments, professors, party venues, and job candidates. Each individual was provided with different features relevant to the problem. In the team discussion phase, participants synchronously chatted with other teammates to share information that they held in order to achieve the optimal ranking. The group communicated through text-based and communication lasted for 20 minutes.
4.3 CPS Skills: Qualitative Coding
|CPS skill code||Definition||Examples|
|Social||Maintaining Communication (SMC)||Off-Topic Communication, Rapport Building Communication, Inappropriate Communication||“nice job guys” “no problem”|
|Sharing Information (SSI)||Share Own Information, Share Task or Resource Information, Share Understanding||“candidate A was listed as having good leadership skills”|
|Establish Shared Understanding (SESU)||Presentation Phase, Acceptance Phase||“What skills do we need?”|
|Negotiating (SN)||Express agreement or disagreement, Resolve conflicts||“You’re right” “My list shows that candidate C is unwilling to further their education.”|
|Cognitive||Representing and Formulating (CRF)||Represent the problem using words, Proposes specific conceptual thinking||“Yeah I feel that B is the best because everything is nearby and the landlord offers a 24-hour maintenance service”|
|Planning (CP)||Set Goals, Develop Strategies||“we have to choose between a and B for being the best”|
|Executing (CE)||Suggesting an action to a teammate, Report of own action||“Please list all your features for candidate C”|
|Monitoring (CM)||Monitor progress toward the goal, Monitor whether teammates are present||“so we in agreement to make B the best?”|
In an attempt to qualitatively annotate the utterance data, we adapted the CPS framework from . We removed one cognitive skill code, exploring and understanding, given that it was not as eminent in our data set and was less relevant to our CPS task. The resulting CPS skills are divided into social and cognitive interactions. Each social and cognitive category includes four CPS skill codes, resulting in eight skill codes in total. Social skills include maintaining communication (SMC), sharing information (SSI), establishing shared understanding (SESU), and negotiating (SN); while cognitive skills consist of representing and formulating (CRF), planning (CP), executing (CE), and monitoring (CM). Table 1 present the definitions and examples of each CPS skill.
CPS skill coding was completed at the chat utterances level and each utterance was assigned one primary code (i.e. eight CPS skill codes aforementioned) and 29 subskills that correspond to each high-level CPS skill. For the purpose of this study, we only focused on the eight main CPS skills. Four undergraduate research assistants were trained as raters to coded the content of students’ discourse (7,711 total utterance events). Raters were trained on the adapted CPS framework. Then, we retrieved a random sample of 20% of all utterances in the data and assigned each rater to code independently. All raters discussed their codes and address any discrepancies. The inter-rater reliability (Kappa = .81) achieved among all raters is considered high (Kappa > .60; ). Next, the remaining 80% of the data were split evenly into four groups. One of the four trained raters coded each of these four groups independently.
4.4 Recurrence quantification analysis
We used recurrence quantification analysis (RQA) to visualize and quantitatively assess students’ behaviors within a CPS environment. Specifically, we used PyRQA Python framework1  to run RQA experiments efficiently. Exploring the recurrence of human behavior would help us better understand how underlying communication patterns occur, and how the phases and dynamics of a system change over time. The time series data required for RQA can be categorical or continuous.
4.4.1 Recurrence plot
Recurrence plot gives us a visual two-dimensional representation to discover repetition, recurrence, and underlying patterns across time from a one-dimensional time series data. For a time-series data of length , the recurrence plot is a matrix consisting of 0, 1 values, where = 1 is an indication of recurrence between and (). An example of this process is provided in Figure 1.
4.4.2 Recurrence quantification
Cultural diversity degree
|Determinism||0.52 (0.2)||0.5 (0.15)||0.46 (0.15)||0.56 (0.12)||0.49 (0.15)|
|Divergence||0.22 (0.1)||0.22 (0.16)||0.24 (0.17)||0.12 (0.06)||0.22 (0.16)|
|Entropy||0.96 (0.62)||0.92 (0.47)||0.81 (0.34)||1.22 (0.44)||0.9 (0.43)|
|Laminarity||0.71 (0.15)||0.67 (0.14)||0.66 (0.13)||0.72 (0.11)||0.67 (0.13)|
|Longest diagonal line||6.29 (4.07)||6.41 (4.28)||5.62 (2.93)||11.24 (6.06)||6.48 (4.21)|
|Recurrence rate||0.24 (0.06)||0.25 (0.05)||0.25 (0.06)||0.25 (0.06)||0.25 (0.06)|
|Discussion Time||715.15 (183.38)||727.34 (358.15)||885.42 (431.21)||1141.0 (285.84)||850.47 (404.31)|
|Average diagonal line||2.9 (1.23)||2.72 (0.73)||2.44 (0.37)||3.2 (1.25)||2.63 (0.74)|
|Number of messages||39.86 (11.02)||49.96 (31.14)||59.24(31.93)||77.47 (27.38)||57.0 (31.69)|
notes: = Mean, = Standard Deviation
Visually, the recurrence plot provides valuable qualitative information about the group dynamics and the structure of the dynamical system. However, RQA’s quantified measures, calculated based on the recurrence plot, allow us to quantitatively evaluate a dynamical system beyond visuals and qualitative observations. The following provides a brief description of the main metrics of RQA (for more information on RQA measures see [78, 47]).
Recurrence Rate: Recurrence rate (RR) shows the rate and the density of recurrence points in a recurrence plot. It is calculated by dividing the recurrence points by the total number of cells in the plot which is the length of the time series squared. Higher recurrence rate would show a higher frequency of repetition in actions and the system to revisit previous states. RR ranges from to , while shows a system without any recurrences, and shows a fully recurrent system.(1)
Determinism: Determinism measures the distribution of recurrence points that form a diagonal line. In other words, determinism is the percentage of recurrence points that align on a diagonal line. The more recurrence points that align on a diagonal line in a recurrence plot, the higher the determinism. As seen in Figure 1, we have four diagonal lines (two diagonal lines of length 3, and two diagonal lines of length 2) we have a fully deterministic system with the determinism of 1. A system with a higher determinism is is considered ordered and repetitious with periodic patterns over the time. A system with a lower determinism can be a sign of a more chaotic system. To compute determinism, needs to be considered as a minimum length of which diagonal lines to consider. For instance, a would eliminate the two diagonal lines of length 2 in Figure 1 and therefore lower the determinism compared to the default value ().(2)
Laminarity: First introduced by , laminarity is designed to capture the percentage of recurrence points that align on a vertical line. A continuation of the same event in a system forms a vertical line. In our study, same skill codes appearing in consecutive messages results in a vertical line. For the computation, a minimum line length of needs to be set to possibly restrict the length of vertical lines in the calculation of laminarity.(3)
Entropy of diagonal lines: Entropy reflects the complexity of the length of diagonal lines in a recurrence plot. It is calculated using the Shannon entropy  of distribution, where is the probability to find a diagonal line of length in the recurrence plot. High entropy is an indication of high variation in length of recurrent sequences. In contrast, in a recurrence plot where all diagonal lines are the same length, the entropy would be zero.(4)
Longest Diagonal Line and Divergence: Another measure related to diagonal lines are (1) the longest diagonal line, , excluding the line of identity, and (2) divergence which is the reverse of . Along with determinism, these two measures are indicators of convergence, chaos, and stability in the system. The lower the , the higher the divergence, chaos, and instability within the dynamical system.(5)
4.5 Data Processing
4.5.1 RQA time series
To prepare the data for RQA, we created time series data using the skill codes associated with each chat within the same conversation. The utterances were grouped together based on the team identification code and therefore each group conversation is represented by a time series of messages and their associated skill codes. Categorical skill codes were mapped to a numeric code that represents that skill throughout the analysis. The mapping is one-to-one meaning that each skill is only mapped to one number and each number only represents one skill. RQA was then applied to the time series of numeric skill codes to explore structural patterns within the group conversation.
4.5.2 Cultural diversity
We quantified cultural diversity based on the heterogeneity of ethnic identities. To determine cultural diversity level, we calculated how many unique ethnicities existed in each group according to the students self-reported demographic information. Since groups consist of four members, there were four possible levels of group diversity ranging from fully homogeneous groups (coded as 1) to fully heterogeneous groups (coded as 4). Below is a breakdown of group compositions and their associated degrees of cultural diversity:
1: (4 White members), N = 7;
2: (3 White members, 1 Asian member) or (2 White members, 2 Asian members), N = 42;
3: (2 White members, 1 Asian member, 1 Hispanic member), N = 67;
4: (1 White member, 1 Asian member, 1 Hispanic member, 1 Black member), N = 13
4.5.3 Statistical Analysis
RQA was applied to a time series of CPS skill codes associated with each message in a conversation. As described in Section 4.4, RQA allowed us to study team dynamics, underlying behavioral patterns, and their complexity. In order to examine the influence of cultural diversity on CPS communication dynamics, we performed a Kruskal-Wallis test on each RQA measure to see whether structural components of dialogues were different across groups. In addition to seven RQA measures described, we also included the total amount of time (seconds) that students spent discussing their decision and the number of messages sent within that discussion time as conversational measures. We also conducted pairwise Wilcoxon rank tests as a post-hoc analysis with Benjamini-Hochberg p-value adjustment to further locate where the significant difference specifically resides. This analysis enabled us to detect significance of variations of conversational measures and their relation to the level of team diversity. The code and results of the RQA Analysis is available on Github at: github.com/The-Language-and-Learning-Analytics-Lab/cult-div-rqa
Descriptive statistics of conversational measures by cultural diversity is available in Table 2. Results for each Kruskal-Wallis tests on Table 3 suggest significant effect of cultural diversity on entropy at [H(3) = 9.077, , = 0.049], longest diagonal line at [H(3) = 14.405, , = 0.091], average diagonal line at [H(3) = 10.858, , = 0.063], longest diagonal line and divergence at [H(3) = 14.405, , = 0.091], number of messages sent at [H(3) = 11.231, , = 0.066]), and the time spent in discussion at [H(3) = 12.334, , = 0.075]). Notably, due to the relationship of divergence and longest diagonal line in Equation , the same results for these two measures were expected. Combined, these suggest structural differences in conversations across the four groups, in terms of complexity of activity and patterns of behavior. We followed up this significant result with pairwise Wilcoxon rank tests reported in Table 4.
|Entropy of Diagonal Lines||3||9.077||0.049|
|Average Diagonal Line||3||10.858||0.063|
|Longest Diagonal Line||3||14.405||0.091|
|Number of Messages||3||11.231||0.066|
notes: = Chi-Squared, df = degrees of freedom
, , ,
The post-hoc analysis suggests significant differences between the most diverse group and the rest of the diversity groups in most occasions, with the most difference apparent in time spent on discussions and longest diagonal line. The most culturally diverse groups spent more time (in seconds) discussing the problem (M = 1141, SD = 285.84) than all three other culturally demographic compositions (Level 1 diversity, M = 715.15, SD = 183.38; Level 2 diversity, M = 727.34, SD = 358.15; Level 3 diversity, M = 885.42, SD = 431.21). Descriptive statistics of the measures grouped by degrees of cultural diversity are available in Table 2. These findings further demonstrate the impact of cultural diversity on the group dynamics and key components of group discussions. Figure 3 visualizes the linear relationship between the RQA measures explored in the post-hoc analysis and the amount of cultural diversity in each group. Of the five RQA measures examined, we observed a steady increase in discussion time, number of messages, and longest diagonal line as degree of cultural diversity increased. In addition, both entropy and average diagonal line increased at a less consistent rate. Specifically, entropy and average diagonal line increased mostly between the second most culturally diverse groups and the most culturally diverse groups. We discuss the implications of the results in the next section.
Average diagonal line
Longest diagonal line
Number of Messages
CPS has been increasingly recognized as an essential 21st century skill in both educational and workplace environments [22, 12], especially in settings where teams are becoming increasingly more diverse and international. As such, exploring ways to further our understanding of the communication behavior patterns in diverse teams and advance CPS skills has been at the forefront of educational research [12, 29]. In this study, we extended current literature that measures performance outcomes by using a dynamical systems lens to examine how the level of in-group diversity influences team communication behavior. Specifically, we quantified CPS skills exhibited in group discourse and characterized conversational structures through RQA measures.
In response to RQ1, we found that degrees of cultural diversity in teams are associated with systematic outcomes of group communication captured through RQA measures. Specifically, these outcomes include number of messages, time spent in chat, complexity and unpredictability of recurrent patterns (entropy), average length of recurrent pattern (average diagonal line in recurrence plot), and longest recurrent pattern (longest diagonal line in recurrence plot). Regarding RQ2, our post-hoc analysis further reveal that groups with the highest degree of cultural diversity had diagnostically different structural patterns of communication compared to more homogeneous teams. Specifically, groups with the highest level of heterogeneity spent more time and sent more messages in chat, had greater complexity in interpersonal exchange, and lower divergence in group discourse. Notably, there was no sign of significant difference among the moderately diverse groups and homogeneous groups. We discuss the implications of results in greater details as follow.
First, the finding that groups with greater culturally diversity spent more time and sent more messages during the discussion could indicate positive or negative group dynamics. On the positive side, higher number of messages signals more opportunities for information sharing and exchange of ideas, which may suggest that members in these teams are more actively engaged in the collaborative discourse and provided their diverse perspectives to the problem. On the other hand, it might also indicate teams getting stuck or struggle to reach consensus. Teams that struggle with internal conflicts also tend to spend more time on communication to resolve disagreements. This issue is more signified in the context of a CPS ranking task. Greater cultural differences in teams might result in different preferences and priorities over different job candidates, apartments, party venue, and professor qualities . As such, increased cultural diversity in teams could bring challenges in team communication that are reflected through frequencies of messages and time spent on discussion. This finding is consistent with the notion that cultural diversity in teams can be a “double-edged sword". It has been widely suggested that cultural diversity is associated with both positive and negative team outcomes . Previous studies demonstrate that compared to culturally homogeneous groups, culturally diverse teams are associated with higher levels of innovation and creativity but are also prone to more conflicts, less effective communication, and less cohesion . It is worth noting that the interpretation of the results has to be situated in the context of tasks. For instance, in open-ended tasks like brainstorming, longer time spent and more messages are typically indications of more innovative perspectives and creativity. However, in decision making tasks, especially when groups were under time constraints to arrive at a decision, more time spent on the task and greater amount of discourse may suggest it takes more effort for heterogeneous groups to build shared understanding and reach a solution collectively.
Our RQA analysis further revealed more culturally diverse teams were associated with more complexity in the distribution of recurrent patterns (higher entropy), longer recurrent patterns of interactions (higher average length of diagonal lines, and higher longest diagonal line). The increased complexity in culturally diverse teams can signal less rigidity, more adaptability, and higher responsivity in team interactions. These attributes have been found to be key for innovation, creativity, and information sharing in team communication . For instance, Fusaroli and Tylén found entropy and average diagonal line of recurrences in transcripts between the dyads to have a positive association with their performance . In conjunction with our first finding, longer discussion time and higher number of messages may provide more opportunities for diverse groups to display recurrent patterns, which may explain why diverse groups have higher longest recurrent patterns. Interestingly, we found that the most culturally diverse groups have noticeably higher longest recurrent patterns and lower divergence, suggesting higher convergence within group discussion over time. This is in contrast to previous research which has typically associated high cultural diversity with divergent communication behavior . We consider two potential explanations for how higher convergence was reached in team interactions within the most diverse teams. First, the impact of diversity on performance can be mediated through goal-orientation and information elaboration . We hypothesize that longer and more active discussions in more diverse groups allowed much more opportunities for information elaboration which may account for a longer recurrent pattern in diverse teams. Moreover, the goal-oriented nature of CPS decision making task may cultivate more convergent collaborative interaction patterns. Second, higher entropy in more diverse groups indicates the higher complexity in the recurrent patterns. Therefore, although the longest recurrent pattern shows convergence in discussions of more diverse teams, higher entropy suggests less rigidity in the patterns of behavior which allows for less habitual patterns and more flexibility in social and cognitive interaction sequences. These findings indicate a potential increased willingness within cultural diverse teams to engage in active information exchange and leverage different perspectives during collaborative discussions. Taken together, this finding suggests highly diverse groups’ flexibility in communication structure and openness to new information. High convergence indicated their capacity in perspective-taking as well as the elaborated processing of new information. Moreover, higher longest diagonal line could possibly be a sign of lengthier discussion and argumentation to reach a conclusion which could be a result of less effective communication. This could be meaningful characteristics to look for as teams across educational and professional settings are increasingly diverse. Further research would be needed to examine whether such pattern is associated with productive performance and psychological outcomes for participating members.
Our work serves as a starting point for future studies to leverage RQA in establishing the possible positive and negative links between cultural diversity, communication dynamics, and other post-collaboration measures such as performance and team satisfaction. Moreover, we contribute a structural view of the recurrences of CPS skills through sociocognitive processes in discourse. By taking a look into how the temporal sociocognitive processes that occur during collaborative interactions are shaped by cultural diversity during CPS tasks, we can begin to take a step towards understanding factors in learning environments that make CPS communication more or less effective. In addition to leveraging CPS frameworks such as  to characterize group discourse, future research may also operationalize group dynamics by capturing linguistic features based on the content of messages through syntactic and semantic similarity, i.e., Conceptual Recurrence Plots . This analysis would not only allow for more complexity and possibilities in the way recurrence appears in group communication, but also potentially reveal more nuanced linguistic patterns with respect to cultural diversity in teams. Finally, our current study mainly concerns with the degree of cultural diversity, however there is also a need to investigate the communication behavior of specific demographic subgroups (i.e. female, underrepresented minorities) in CPS. Further investigation on the differences within these demographic groups could examine how these factors independently and in combination play a role in shaping diverse groups. We call for more efforts towards promoting inclusivity and equity in teams. Future studies along this line should aim to provide further insights on identifying group patterns that promote effective problem solving and meaningful experiences among diverse teams.
We focused on the impact of cultural diversity on group communication in CPS. Our novel approach leveraged RQA to study the dynamics in group communication. Applying RQA to analyze CPS discourse provides a means to uncover how groups with different degrees of diversity exhibit different patterns of behavior during the collaborative process. Understanding these behaviors is essential given the dynamic and interdependent nature of CPS tasks. Specifically, insight on how group dynamics impact culturally diverse groups has important implications on monitoring and improving diverse group communications. In sum, our study emphasizes the need to further our understanding of the role of diversity in group communication behaviors as teams share information, negotiate, and navigate the problem-solving task. Exploring the intricacies of these group dynamics sets a first step towards future research on understanding how these behaviors relate to group outcomes such as performance, psychological experiences, and group satisfaction. Furthermore, this study aligns with the agenda of promoting diversity and inclusivity in AI systems. Our findings could be meaningful for the researchers in the greater EDM community for applying such methods to diagnose diverse team dynamics, as well as further inquiring positioning AI’s critical role in structurally complex collaborative processes across diverse teams.
This research was supported by The Gates Foundation (#INV - 000752) and The Andrew W. Mellon Foundation (1806-05902). The authors would like to thank the Next Generation Undergraduate Success Measurement Project team members for help with data collection. Thanks to Educational Testing Services for the experiment information.
- J. J. Andrews and D. N. Rapp. Benefits, costs, and challenges of collaboration for learning and memory. Translational Issues in Psychological Science, 1(2):182, 2015.
- J. Andrews-Todd and C. M. Forsyth. Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Computers in human behavior, 104:105759, 2020.
- D. Angus, A. Smith, and J. Wiles. Conceptual recurrence plots: Revealing patterns in human discourse. IEEE transactions on Visualization and Computer Graphics, 18(6):988–997, 2011.
- D. Angus, A. E. Smith, and J. Wiles. Human communication as coupled time series: Quantifying multi-participant recurrence. IEEE Transactions on Audio, Speech, and Language Processing, 20(6):1795–1807, 2012.
- R. Arum, J. S. Eccles, J. Heckhausen, G. A. Orona, L. von Keyserlingk, C. M. Wegemer, C. E. Wright, and K. Yamaguchi-Pedroza. A framework for measuring undergraduate learning and growth. Change: The Magazine of Higher Learning, 53(6):51–59, 2021.
- J. B. Bear and A. W. Woolley. The role of gender in team collaboration and performance. Interdisciplinary science reviews, 36(2):146–153, 2011.
- S. T. Bell, A. J. Villado, M. A. Lukasik, L. Belau, and A. L. Briggs. Getting specific about demographic diversity variable and team performance relationships: A meta-analysis. Journal of management, 37(3):709–743, 2011.
- C. A. Bowers, J. A. Pharmer, and E. Salas. When member homogeneity is needed in work teams: A meta-analysis. Small group research, 31(3):305–327, 2000.
- J. Burrus, T. Jackson, N. Xi, and J. Steinberg. Identifying the most important 21st century workforce competencies: An analysis of the occupational information network (o* net). ETS Research Report Series, 2013(2):i–55, 2013.
- W. Cade, N. Dowell, A. Graesser, Y. Tausczik, and J. Pennebaker. Modeling student socioaffective responses to group interactions in a collaborative online chat environment. In Educational Data Mining 2014. Citeseer, 2014.
- Z. Cai, B. Eagan, N. Dowell, J. Pennebaker, D. Shaffer, and A. Graesser. Epistemic network analysis and topic modeling for chat data from collaborative learning environment. In Proceedings of the 10th international conference on educational data mining, 2017.
- E. Care, C. Scoular, and P. Griffin. Assessment of collaborative problem solving in education environments. Applied Measurement in Education, 29(4):250–264, 2016.
- P. Chopade, D. Edwards, S. M. Khan, A. Andrade, and S. Pu. Cpsx: Using ai-machine learning for mapping human-human interaction and measurement of cps teamwork skills. In 2019 IEEE International Symposium on Technologies for Homeland Security (HST), pages 1–6. IEEE, 2019.
- T. Cox. Cultural diversity in organizations: Theory, research and practice. Berrett-Koehler Publishers, 1994.
- R. Dale, R. Fusaroli, N. D. Duran, and D. C. Richardson. The self-organization of human interaction. In Psychology of learning and motivation, volume 59, pages 43–95. Elsevier, 2013.
- N. Dowell, Y. Lin, A. Godfrey, and C. Brooks. Promoting inclusivity through time-dynamic discourse analysis in digitally-mediated collaborative learning. In International Conference on Artificial Intelligence in Education, pages 207–219. Springer, 2019.
- N. M. Dowell, W. L. Cade, Y. Tausczik, J. Pennebaker, and A. C. Graesser. What works: Creating adaptive and intelligent systems for collaborative learning support. In International conference on intelligent tutoring systems, pages 124–133. Springer, 2014.
- N. M. Dowell, Y. Lin, A. Godfrey, and C. Brooks. Exploring the relationship between emergent sociocognitive roles, collaborative problem-solving skills, and outcomes: A group communication analysis. Journal of Learning Analytics, 7(1):38–57, 2020.
- N. M. Dowell, T. M. Nixon, and A. C. Graesser. Group communication analysis: A computational linguistics approach for detecting sociocognitive roles in multiparty interactions. Behavior research methods, 51(3):1007–1041, 2019.
- N. M. Dowell and O. Poquet. Scip: Combining group communication and interpersonal positioning to identify emergent roles in scaled digital environments. Computers in Human Behavior, 119:106709, 2021.
- R. J. Ely and D. A. Thomas. Cultural diversity at work: The effects of diversity perspectives on work group processes and outcomes. Administrative science quarterly, 46(2):229–273, 2001.
- S. M. Fiore, A. Graesser, and S. Greiff. Collaborative problem-solving education for the twenty-first-century workforce. Nature human behaviour, 2(6):367–369, 2018.
- S. M. Fiore, K. A. Smith-Jentsch, E. Salas, N. Warner, and M. Letsky. Towards an understanding of macrocognition in teams: developing and defining complex collaborative processes and products. Theoretical Issues in Ergonomics Science, 11(4):250–271, 2010.
- E. G. Foldy. Learning from diversity: A theoretical exploration. Public Administration Review, 64(5):529–538, 2004.
- R. Fusaroli and K. Tylén. Investigating conversational dynamics: Interactive alignment, interpersonal synergy, and collective task performance. Cognitive science, 40(1):145–171, 2016.
- J. C. Gorman, N. J. Cooke, P. G. Amazeen, and S. Fouse. Measuring patterns in team interaction sequences using a discrete recurrence approach. Human Factors, 54(4):503–517, 2012.
- A. C. Graesser, N. Dowell, and D. Clewley. Assessing collaborative problem solving through conversational agents. In Innovative assessment of collaboration, pages 65–80. Springer, 2017.
- A. C. Graesser, N. Dowell, A. J. Hampton, A. M. Lippert, H. Li, and D. W. Shaffer. Building intelligent conversational tutors and mentors for team collaborative problem solving: Guidance from the 2015 program for international student assessment. In Building intelligent tutoring systems for teams. Emerald Publishing Limited, 2018.
- A. C. Graesser, S. M. Fiore, S. Greiff, J. Andrews-Todd, P. W. Foltz, and F. W. Hesse. Advancing the science of collaborative problem solving. Psychological Science in the Public Interest, 19(2):59–92, 2018.
- J. Hao, L. Liu, A. A. von Davier, N. Lederer, D. Zapata-Rivera, P. Jakl, and M. Bakkenson. Epcal: Ets platform for collaborative assessment and learning. ETS Research Report Series, 2017(1):1–14, 2017.
- D. A. Harrison, K. H. Price, and M. P. Bell. Beyond relational demography: Time and the effects of surface-and deep-level diversity on work group cohesion. Academy of management journal, 41(1):96–107, 1998.
- A. C. Homan, J. R. Hollenbeck, S. E. Humphrey, D. V. Knippenberg, D. R. Ilgen, and G. A. Van Kleef. Facing differences with an open mind: Openness to experience, salience of intragroup differences, and performance of diverse work groups. Academy of Management Journal, 51(6):1204–1222, 2008.
- S. K. Horwitz and I. B. Horwitz. The effects of team diversity on team outcomes: A meta-analytic review of team demography. Journal of management, 33(6):987–1015, 2007.
- H. Jeong and M. T. Chi. Knowledge convergence and collaborative learning. Instructional Science, 35(4):287–315, 2007.
- H. Jeong, C. E. Hmelo-Silver, and K. Jo. Ten years of computer-supported collaborative learning: A meta-analysis of cscl in stem education during 2005–2014. Educational research review, 28:100284, 2019.
- A. P. Knight, D. M. Kennedy, and S. A. McComb. Using recurrence analysis to examine group dynamics. Group Dynamics: Theory, Research, and Practice, 20(3):223, 2016.
- A. Kramer, D. P. Bhave, and T. D. Johnson. Personality and group performance: The importance of personality composition and work tasks. Personality and Individual Differences, 58:132–137, 2014.
- P. C. Kyllonen, M. Zhu, and A. A. v. Davier. Innovative assessment of collaboration. Springer, 2017.
- J. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. biometrics, pages 159–174, 1977.
- S. S. Levine, E. P. Apfelbaum, M. Bernard, V. L. Bartelt, E. J. Zajac, and D. Stark. Ethnic diversity deflates price bubbles. Proceedings of the National Academy of Sciences, 111(52):18524–18529, 2014.
- Y. Lin, N. Dowell, and A. Godfrey. Skills matter: Modeling the relationship between decision making processes and collaborative problem-solving skills during hidden profile tasks. In LAK21: 11th International Learning Analytics and Knowledge Conference, pages 428–437, 2021.
- Y. Lin, N. Dowell, A. Godfrey, H. Choi, and C. Brooks. Modeling gender dynamics in intra and interpersonal interactions during online collaborative learning. In Proceedings of the 9th international conference on learning analytics & knowledge, pages 431–435, 2019.
- L. Liu, J. Hao, A. A. von Davier, P. Kyllonen, and J.-D. Zapata-Rivera. A tough nut to crack: Measuring collaborative problem solving. In Handbook of research on technology tools for real-world skill development, pages 344–359. IGI Global, 2016.
- Y. Lou, P. C. Abrami, and S. d’Apollonia. Small group and individual learning with technology: A meta-analysis. Review of educational research, 71(3):449–521, 2001.
- L. Lu, Y. C. Yuan, and P. L. McLeod. Twenty-five years of hidden profiles in group decision making: A meta-analysis. Personality and Social Psychology Review, 16(1):54–75, 2012.
- M. A. Marks, J. E. Mathieu, and S. J. Zaccaro. A temporally based framework and taxonomy of team processes. Academy of management review, 26(3):356–376, 2001.
- N. Marwan, M. Carmen Romano, M. Thiel, and J. Kurths. Recurrence plots for the analysis of complex systems. Physics Reports, 438(5):237–329, 2007.
- N. Marwan, N. Wessel, U. Meyerfeldt, A. Schirdewan, and J. Kurths. Recurrence-plot-based measures of complexity and their application to heart-rate-variability data. Physical review E, 66(2):026702, 2002.
- D. McGunagle and L. Zizka. Employability skills for 21st-century stem students: the employers’ perspective. Higher Education, Skills and Work-Based Learning, 2020.
- P. L. McLeod, S. A. Lobel, and T. H. Cox Jr. Ethnic diversity and creativity in small groups. Small group research, 27(2):248–264, 1996.
- A. Nederveen Pieterse, D. Van Knippenberg, and D. Van Dierendonck. Cultural diversity and team performance: The role of team member goal orientation. Academy of management journal, 56(3):782–804, 2013.
- OECD. Pisa 2015 collaborative problem solving framework, 2013.
- P. Paulus. Groups, teams, and creativity: The creative potential of idea-generating groups. Applied psychology, 49(2):237–262, 2000.
- V. Popov, O. Noroozi, J. B. Barrett, H. J. Biemans, S. D. Teasley, B. Slof, and M. Mulder. Perceptions and experiences of, and outcomes for, university students in culturally diversified dyads in a computer-supported collaborative learning environment. Computers in Human Behavior, 32:186–200, 2014.
- O. Poquet, N. Dowell, C. Brooks, and S. Dawson. Are mooc forums changing? In Proceedings of the 8th International Conference on Learning Analytics and Knowledge, pages 340–349, 2018.
- T. Rawald. Scalable and Efficient Analysis of Large High-Dimensional Data Sets in the Context of Recurrence Analysis. PhD thesis, Humboldt-Universität zu Berlin, Mathematisch-Naturwissenschaftliche Fakultät, 2018.
- T. Reimer, A. Reimer, and U. Czienskowski. Decision-making groups attenuate the discussion bias in favor of shared information: A meta-analysis. Communication Monographs, 77(1):121–142, 2010.
- D. C. Richardson, R. Dale, and N. Z. Kirkham. The art of conversation is coordination. Psychological science, 18(5):407–413, 2007.
- M.-É. Roberge and R. Van Dick. Recognizing the benefits of diversity: When and how does diversity increase group performance? Human resource management review, 20(4):295–308, 2010.
- M. Sarkar, T. Overton, C. Thompson, and G. Rayner. Graduate employability: Views of recent science graduates and employers. International Journal of Innovation in Science and Mathematics Education, 24(3), 2016.
- M. C. Schippers, A. C. Edmondson, and M. A. West. Team reflexivity as an antidote to team information-processing failures. Small Group Research, 45(6):731–769, 2014.
- B. Schneider, N. Dowell, and K. Thompson. Collaboration analytics—current state and potential futures. Journal of Learning Analytics, 8(1):1–12, 2021.
- C. E. Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948.
- S. R. Sommers. On racial diversity and group decision making: identifying multiple effects of racial composition on jury deliberations. Journal of personality and social psychology, 90(4):597, 2006.
- G. K. Stahl and M. L. Maznevski. Unraveling the effects of cultural diversity in teams: A retrospective of research on multicultural work groups and an agenda for future research. Journal of International Business Studies, 52(1):4–22, 2021.
- G. K. Stahl, M. L. Maznevski, A. Voigt, and K. Jonsen. Unraveling the effects of cultural diversity in teams: A meta-analysis of research on multicultural work groups. Journal of international business studies, 41(4):690–709, 2010.
- A. J. Strang, G. J. Funke, S. M. Russell, A. W. Dukes, and M. S. Middendorf. Physio-behavioral coupling in a cooperative team task: contributors and relations. Journal of Experimental Psychology: Human Perception and Performance, 40(1):145, 2014.
- Y. Tausczik and X. Huang. The impact of group size on the discovery of hidden profiles in online discussion groups. ACM Transactions on Social Computing, 2(3):1–25, 2019.
- H. Tenzer, M. Pudelko, and A.-W. Harzing. The impact of language barriers on trust formation in multinational teams. Journal of International Business Studies, 45(5):508–535, 2014.
- D. C. Thomas. Cultural diversity and work group effectiveness: An experimental study. Journal of cross-cultural psychology, 30(2):242–263, 1999.
- C. Tröster, A. Mehra, and D. van Knippenberg. Structuring for team success: The interactive effects of network structure and cultural diversity on team potency and performance. Organizational Behavior and Human Decision Processes, 124(2):245–255, 2014.
- H. Van Dijk, M. L. Van Engen, and D. Van Knippenberg. Defying conventional wisdom: A meta-analytical examination of the differences between demographic and job-related diversity relationships with performance. Organizational Behavior and Human Decision Processes, 119(1):38–53, 2012.
- M. Vigier and H. Spencer-Oatey. The interplay of rules, asymmetries in language fluency, and team dynamics in culturally diverse teams: Case study insights. Cross Cultural & Strategic Management, 2018.
- A. A. von Davier. Computational psychometrics in support of collaborative educational assessments, 2017.
- J. Wang, G. H.-L. Cheng, T. Chen, and K. Leung. Team creativity/innovation in culturally diverse teams: A meta-analysis. Journal of Organizational Behavior, 40(6):693–708, 2019.
- W. E. Watson, K. Kumar, and L. K. Michaelsen. Cultural diversity’s impact on interaction process and performance: Comparing homogeneous and diverse task groups. Academy of management journal, 36(3):590–602, 1993.
- S. S. Webber and L. M. Donahue. Impact of highly and less job-related diversity on work group cohesion and performance: A meta-analysis. Journal of management, 27(2):141–162, 2001.
- C. L. Webber Jr and J. P. Zbilut. Dynamical assessment of physiological systems and states using recurrence plot strategies. Journal of applied physiology, 76(2):965–973, 1994.
- C. L. Webber Jr and J. P. Zbilut. Recurrence quantification analysis of nonlinear dynamical systems. Tutorials in contemporary nonlinear methods for the behavioral sciences, 94(2005):26–94, 2005.
- R. Whorton, A. Casillas, F. L. Oswald, A. Shaw, J. Burrus, K. Mattern, B. Naemi, and R. Roberts. Critical skills for the 21st century workforce. Building better students: Preparation for the workforce, pages 47–72, 2017.
- A. W. Woolley, C. F. Chabris, A. Pentland, N. Hashmi, and T. W. Malone. Evidence for a collective intelligence factor in the performance of human groups. science, 330(6004):686–688, 2010.
© 2022 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.