research on problem solving skills

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 11 January 2023

The effectiveness of collaborative problem solving in promoting students’ critical thinking: A meta-analysis based on empirical literature

• Enwei Xu   ORCID: orcid.org/0000-0001-6424-8169 1 ,
• Wei Wang 1 &
• Qingxia Wang 1  

Humanities and Social Sciences Communications volume  10 , Article number:  16 ( 2023 ) Cite this article

19k Accesses

21 Citations

3 Altmetric

Metrics details

• Science, technology and society

Collaborative problem-solving has been widely embraced in the classroom instruction of critical thinking, which is regarded as the core of curriculum reform based on key competencies in the field of education as well as a key competence for learners in the 21st century. However, the effectiveness of collaborative problem-solving in promoting students’ critical thinking remains uncertain. This current research presents the major findings of a meta-analysis of 36 pieces of the literature revealed in worldwide educational periodicals during the 21st century to identify the effectiveness of collaborative problem-solving in promoting students’ critical thinking and to determine, based on evidence, whether and to what extent collaborative problem solving can result in a rise or decrease in critical thinking. The findings show that (1) collaborative problem solving is an effective teaching approach to foster students’ critical thinking, with a significant overall effect size (ES = 0.82, z  = 12.78, P  < 0.01, 95% CI [0.69, 0.95]); (2) in respect to the dimensions of critical thinking, collaborative problem solving can significantly and successfully enhance students’ attitudinal tendencies (ES = 1.17, z  = 7.62, P  < 0.01, 95% CI[0.87, 1.47]); nevertheless, it falls short in terms of improving students’ cognitive skills, having only an upper-middle impact (ES = 0.70, z  = 11.55, P  < 0.01, 95% CI[0.58, 0.82]); and (3) the teaching type (chi 2  = 7.20, P  < 0.05), intervention duration (chi 2  = 12.18, P  < 0.01), subject area (chi 2  = 13.36, P  < 0.05), group size (chi 2  = 8.77, P  < 0.05), and learning scaffold (chi 2  = 9.03, P  < 0.01) all have an impact on critical thinking, and they can be viewed as important moderating factors that affect how critical thinking develops. On the basis of these results, recommendations are made for further study and instruction to better support students’ critical thinking in the context of collaborative problem-solving.

Similar content being viewed by others

A meta-analysis of the effects of design thinking on student learning

Fostering twenty-first century skills among primary school students through math project-based learning

A meta-analysis to gauge the impact of pedagogies employed in mixed-ability high school biology classrooms

Introduction.

Although critical thinking has a long history in research, the concept of critical thinking, which is regarded as an essential competence for learners in the 21st century, has recently attracted more attention from researchers and teaching practitioners (National Research Council, 2012 ). Critical thinking should be the core of curriculum reform based on key competencies in the field of education (Peng and Deng, 2017 ) because students with critical thinking can not only understand the meaning of knowledge but also effectively solve practical problems in real life even after knowledge is forgotten (Kek and Huijser, 2011 ). The definition of critical thinking is not universal (Ennis, 1989 ; Castle, 2009 ; Niu et al., 2013 ). In general, the definition of critical thinking is a self-aware and self-regulated thought process (Facione, 1990 ; Niu et al., 2013 ). It refers to the cognitive skills needed to interpret, analyze, synthesize, reason, and evaluate information as well as the attitudinal tendency to apply these abilities (Halpern, 2001 ). The view that critical thinking can be taught and learned through curriculum teaching has been widely supported by many researchers (e.g., Kuncel, 2011 ; Leng and Lu, 2020 ), leading to educators’ efforts to foster it among students. In the field of teaching practice, there are three types of courses for teaching critical thinking (Ennis, 1989 ). The first is an independent curriculum in which critical thinking is taught and cultivated without involving the knowledge of specific disciplines; the second is an integrated curriculum in which critical thinking is integrated into the teaching of other disciplines as a clear teaching goal; and the third is a mixed curriculum in which critical thinking is taught in parallel to the teaching of other disciplines for mixed teaching training. Furthermore, numerous measuring tools have been developed by researchers and educators to measure critical thinking in the context of teaching practice. These include standardized measurement tools, such as WGCTA, CCTST, CCTT, and CCTDI, which have been verified by repeated experiments and are considered effective and reliable by international scholars (Facione and Facione, 1992 ). In short, descriptions of critical thinking, including its two dimensions of attitudinal tendency and cognitive skills, different types of teaching courses, and standardized measurement tools provide a complex normative framework for understanding, teaching, and evaluating critical thinking.

Cultivating critical thinking in curriculum teaching can start with a problem, and one of the most popular critical thinking instructional approaches is problem-based learning (Liu et al., 2020 ). Duch et al. ( 2001 ) noted that problem-based learning in group collaboration is progressive active learning, which can improve students’ critical thinking and problem-solving skills. Collaborative problem-solving is the organic integration of collaborative learning and problem-based learning, which takes learners as the center of the learning process and uses problems with poor structure in real-world situations as the starting point for the learning process (Liang et al., 2017 ). Students learn the knowledge needed to solve problems in a collaborative group, reach a consensus on problems in the field, and form solutions through social cooperation methods, such as dialogue, interpretation, questioning, debate, negotiation, and reflection, thus promoting the development of learners’ domain knowledge and critical thinking (Cindy, 2004 ; Liang et al., 2017 ).

Collaborative problem-solving has been widely used in the teaching practice of critical thinking, and several studies have attempted to conduct a systematic review and meta-analysis of the empirical literature on critical thinking from various perspectives. However, little attention has been paid to the impact of collaborative problem-solving on critical thinking. Therefore, the best approach for developing and enhancing critical thinking throughout collaborative problem-solving is to examine how to implement critical thinking instruction; however, this issue is still unexplored, which means that many teachers are incapable of better instructing critical thinking (Leng and Lu, 2020 ; Niu et al., 2013 ). For example, Huber ( 2016 ) provided the meta-analysis findings of 71 publications on gaining critical thinking over various time frames in college with the aim of determining whether critical thinking was truly teachable. These authors found that learners significantly improve their critical thinking while in college and that critical thinking differs with factors such as teaching strategies, intervention duration, subject area, and teaching type. The usefulness of collaborative problem-solving in fostering students’ critical thinking, however, was not determined by this study, nor did it reveal whether there existed significant variations among the different elements. A meta-analysis of 31 pieces of educational literature was conducted by Liu et al. ( 2020 ) to assess the impact of problem-solving on college students’ critical thinking. These authors found that problem-solving could promote the development of critical thinking among college students and proposed establishing a reasonable group structure for problem-solving in a follow-up study to improve students’ critical thinking. Additionally, previous empirical studies have reached inconclusive and even contradictory conclusions about whether and to what extent collaborative problem-solving increases or decreases critical thinking levels. As an illustration, Yang et al. ( 2008 ) carried out an experiment on the integrated curriculum teaching of college students based on a web bulletin board with the goal of fostering participants’ critical thinking in the context of collaborative problem-solving. These authors’ research revealed that through sharing, debating, examining, and reflecting on various experiences and ideas, collaborative problem-solving can considerably enhance students’ critical thinking in real-life problem situations. In contrast, collaborative problem-solving had a positive impact on learners’ interaction and could improve learning interest and motivation but could not significantly improve students’ critical thinking when compared to traditional classroom teaching, according to research by Naber and Wyatt ( 2014 ) and Sendag and Odabasi ( 2009 ) on undergraduate and high school students, respectively.

The above studies show that there is inconsistency regarding the effectiveness of collaborative problem-solving in promoting students’ critical thinking. Therefore, it is essential to conduct a thorough and trustworthy review to detect and decide whether and to what degree collaborative problem-solving can result in a rise or decrease in critical thinking. Meta-analysis is a quantitative analysis approach that is utilized to examine quantitative data from various separate studies that are all focused on the same research topic. This approach characterizes the effectiveness of its impact by averaging the effect sizes of numerous qualitative studies in an effort to reduce the uncertainty brought on by independent research and produce more conclusive findings (Lipsey and Wilson, 2001 ).

This paper used a meta-analytic approach and carried out a meta-analysis to examine the effectiveness of collaborative problem-solving in promoting students’ critical thinking in order to make a contribution to both research and practice. The following research questions were addressed by this meta-analysis:

What is the overall effect size of collaborative problem-solving in promoting students’ critical thinking and its impact on the two dimensions of critical thinking (i.e., attitudinal tendency and cognitive skills)?

How are the disparities between the study conclusions impacted by various moderating variables if the impacts of various experimental designs in the included studies are heterogeneous?

This research followed the strict procedures (e.g., database searching, identification, screening, eligibility, merging, duplicate removal, and analysis of included studies) of Cooper’s ( 2010 ) proposed meta-analysis approach for examining quantitative data from various separate studies that are all focused on the same research topic. The relevant empirical research that appeared in worldwide educational periodicals within the 21st century was subjected to this meta-analysis using Rev-Man 5.4. The consistency of the data extracted separately by two researchers was tested using Cohen’s kappa coefficient, and a publication bias test and a heterogeneity test were run on the sample data to ascertain the quality of this meta-analysis.

Data sources and search strategies

There were three stages to the data collection process for this meta-analysis, as shown in Fig. 1 , which shows the number of articles included and eliminated during the selection process based on the statement and study eligibility criteria.

This flowchart shows the number of records identified, included and excluded in the article.

First, the databases used to systematically search for relevant articles were the journal papers of the Web of Science Core Collection and the Chinese Core source journal, as well as the Chinese Social Science Citation Index (CSSCI) source journal papers included in CNKI. These databases were selected because they are credible platforms that are sources of scholarly and peer-reviewed information with advanced search tools and contain literature relevant to the subject of our topic from reliable researchers and experts. The search string with the Boolean operator used in the Web of Science was “TS = (((“critical thinking” or “ct” and “pretest” or “posttest”) or (“critical thinking” or “ct” and “control group” or “quasi experiment” or “experiment”)) and (“collaboration” or “collaborative learning” or “CSCL”) and (“problem solving” or “problem-based learning” or “PBL”))”. The research area was “Education Educational Research”, and the search period was “January 1, 2000, to December 30, 2021”. A total of 412 papers were obtained. The search string with the Boolean operator used in the CNKI was “SU = (‘critical thinking’*‘collaboration’ + ‘critical thinking’*‘collaborative learning’ + ‘critical thinking’*‘CSCL’ + ‘critical thinking’*‘problem solving’ + ‘critical thinking’*‘problem-based learning’ + ‘critical thinking’*‘PBL’ + ‘critical thinking’*‘problem oriented’) AND FT = (‘experiment’ + ‘quasi experiment’ + ‘pretest’ + ‘posttest’ + ‘empirical study’)” (translated into Chinese when searching). A total of 56 studies were found throughout the search period of “January 2000 to December 2021”. From the databases, all duplicates and retractions were eliminated before exporting the references into Endnote, a program for managing bibliographic references. In all, 466 studies were found.

Second, the studies that matched the inclusion and exclusion criteria for the meta-analysis were chosen by two researchers after they had reviewed the abstracts and titles of the gathered articles, yielding a total of 126 studies.

Third, two researchers thoroughly reviewed each included article’s whole text in accordance with the inclusion and exclusion criteria. Meanwhile, a snowball search was performed using the references and citations of the included articles to ensure complete coverage of the articles. Ultimately, 36 articles were kept.

Two researchers worked together to carry out this entire process, and a consensus rate of almost 94.7% was reached after discussion and negotiation to clarify any emerging differences.

Eligibility criteria

Since not all the retrieved studies matched the criteria for this meta-analysis, eligibility criteria for both inclusion and exclusion were developed as follows:

The publication language of the included studies was limited to English and Chinese, and the full text could be obtained. Articles that did not meet the publication language and articles not published between 2000 and 2021 were excluded.

The research design of the included studies must be empirical and quantitative studies that can assess the effect of collaborative problem-solving on the development of critical thinking. Articles that could not identify the causal mechanisms by which collaborative problem-solving affects critical thinking, such as review articles and theoretical articles, were excluded.

The research method of the included studies must feature a randomized control experiment or a quasi-experiment, or a natural experiment, which have a higher degree of internal validity with strong experimental designs and can all plausibly provide evidence that critical thinking and collaborative problem-solving are causally related. Articles with non-experimental research methods, such as purely correlational or observational studies, were excluded.

The participants of the included studies were only students in school, including K-12 students and college students. Articles in which the participants were non-school students, such as social workers or adult learners, were excluded.

The research results of the included studies must mention definite signs that may be utilized to gauge critical thinking’s impact (e.g., sample size, mean value, or standard deviation). Articles that lacked specific measurement indicators for critical thinking and could not calculate the effect size were excluded.

Data coding design

In order to perform a meta-analysis, it is necessary to collect the most important information from the articles, codify that information’s properties, and convert descriptive data into quantitative data. Therefore, this study designed a data coding template (see Table 1 ). Ultimately, 16 coding fields were retained.

The designed data-coding template consisted of three pieces of information. Basic information about the papers was included in the descriptive information: the publishing year, author, serial number, and title of the paper.

The variable information for the experimental design had three variables: the independent variable (instruction method), the dependent variable (critical thinking), and the moderating variable (learning stage, teaching type, intervention duration, learning scaffold, group size, measuring tool, and subject area). Depending on the topic of this study, the intervention strategy, as the independent variable, was coded into collaborative and non-collaborative problem-solving. The dependent variable, critical thinking, was coded as a cognitive skill and an attitudinal tendency. And seven moderating variables were created by grouping and combining the experimental design variables discovered within the 36 studies (see Table 1 ), where learning stages were encoded as higher education, high school, middle school, and primary school or lower; teaching types were encoded as mixed courses, integrated courses, and independent courses; intervention durations were encoded as 0–1 weeks, 1–4 weeks, 4–12 weeks, and more than 12 weeks; group sizes were encoded as 2–3 persons, 4–6 persons, 7–10 persons, and more than 10 persons; learning scaffolds were encoded as teacher-supported learning scaffold, technique-supported learning scaffold, and resource-supported learning scaffold; measuring tools were encoded as standardized measurement tools (e.g., WGCTA, CCTT, CCTST, and CCTDI) and self-adapting measurement tools (e.g., modified or made by researchers); and subject areas were encoded according to the specific subjects used in the 36 included studies.

The data information contained three metrics for measuring critical thinking: sample size, average value, and standard deviation. It is vital to remember that studies with various experimental designs frequently adopt various formulas to determine the effect size. And this paper used Morris’ proposed standardized mean difference (SMD) calculation formula ( 2008 , p. 369; see Supplementary Table S3 ).

Procedure for extracting and coding data

According to the data coding template (see Table 1 ), the 36 papers’ information was retrieved by two researchers, who then entered them into Excel (see Supplementary Table S1 ). The results of each study were extracted separately in the data extraction procedure if an article contained numerous studies on critical thinking, or if a study assessed different critical thinking dimensions. For instance, Tiwari et al. ( 2010 ) used four time points, which were viewed as numerous different studies, to examine the outcomes of critical thinking, and Chen ( 2013 ) included the two outcome variables of attitudinal tendency and cognitive skills, which were regarded as two studies. After discussion and negotiation during data extraction, the two researchers’ consistency test coefficients were roughly 93.27%. Supplementary Table S2 details the key characteristics of the 36 included articles with 79 effect quantities, including descriptive information (e.g., the publishing year, author, serial number, and title of the paper), variable information (e.g., independent variables, dependent variables, and moderating variables), and data information (e.g., mean values, standard deviations, and sample size). Following that, testing for publication bias and heterogeneity was done on the sample data using the Rev-Man 5.4 software, and then the test results were used to conduct a meta-analysis.

Publication bias test

When the sample of studies included in a meta-analysis does not accurately reflect the general status of research on the relevant subject, publication bias is said to be exhibited in this research. The reliability and accuracy of the meta-analysis may be impacted by publication bias. Due to this, the meta-analysis needs to check the sample data for publication bias (Stewart et al., 2006 ). A popular method to check for publication bias is the funnel plot; and it is unlikely that there will be publishing bias when the data are equally dispersed on either side of the average effect size and targeted within the higher region. The data are equally dispersed within the higher portion of the efficient zone, consistent with the funnel plot connected with this analysis (see Fig. 2 ), indicating that publication bias is unlikely in this situation.

This funnel plot shows the result of publication bias of 79 effect quantities across 36 studies.

Heterogeneity test

To select the appropriate effect models for the meta-analysis, one might use the results of a heterogeneity test on the data effect sizes. In a meta-analysis, it is common practice to gauge the degree of data heterogeneity using the I 2 value, and I 2  ≥ 50% is typically understood to denote medium-high heterogeneity, which calls for the adoption of a random effect model; if not, a fixed effect model ought to be applied (Lipsey and Wilson, 2001 ). The findings of the heterogeneity test in this paper (see Table 2 ) revealed that I 2 was 86% and displayed significant heterogeneity ( P  < 0.01). To ensure accuracy and reliability, the overall effect size ought to be calculated utilizing the random effect model.

The analysis of the overall effect size

This meta-analysis utilized a random effect model to examine 79 effect quantities from 36 studies after eliminating heterogeneity. In accordance with Cohen’s criterion (Cohen, 1992 ), it is abundantly clear from the analysis results, which are shown in the forest plot of the overall effect (see Fig. 3 ), that the cumulative impact size of cooperative problem-solving is 0.82, which is statistically significant ( z  = 12.78, P  < 0.01, 95% CI [0.69, 0.95]), and can encourage learners to practice critical thinking.

This forest plot shows the analysis result of the overall effect size across 36 studies.

In addition, this study examined two distinct dimensions of critical thinking to better understand the precise contributions that collaborative problem-solving makes to the growth of critical thinking. The findings (see Table 3 ) indicate that collaborative problem-solving improves cognitive skills (ES = 0.70) and attitudinal tendency (ES = 1.17), with significant intergroup differences (chi 2  = 7.95, P  < 0.01). Although collaborative problem-solving improves both dimensions of critical thinking, it is essential to point out that the improvements in students’ attitudinal tendency are much more pronounced and have a significant comprehensive effect (ES = 1.17, z  = 7.62, P  < 0.01, 95% CI [0.87, 1.47]), whereas gains in learners’ cognitive skill are slightly improved and are just above average. (ES = 0.70, z  = 11.55, P  < 0.01, 95% CI [0.58, 0.82]).

The analysis of moderator effect size

The whole forest plot’s 79 effect quantities underwent a two-tailed test, which revealed significant heterogeneity ( I 2  = 86%, z  = 12.78, P  < 0.01), indicating differences between various effect sizes that may have been influenced by moderating factors other than sampling error. Therefore, exploring possible moderating factors that might produce considerable heterogeneity was done using subgroup analysis, such as the learning stage, learning scaffold, teaching type, group size, duration of the intervention, measuring tool, and the subject area included in the 36 experimental designs, in order to further explore the key factors that influence critical thinking. The findings (see Table 4 ) indicate that various moderating factors have advantageous effects on critical thinking. In this situation, the subject area (chi 2  = 13.36, P  < 0.05), group size (chi 2  = 8.77, P  < 0.05), intervention duration (chi 2  = 12.18, P  < 0.01), learning scaffold (chi 2  = 9.03, P  < 0.01), and teaching type (chi 2  = 7.20, P  < 0.05) are all significant moderators that can be applied to support the cultivation of critical thinking. However, since the learning stage and the measuring tools did not significantly differ among intergroup (chi 2  = 3.15, P  = 0.21 > 0.05, and chi 2  = 0.08, P  = 0.78 > 0.05), we are unable to explain why these two factors are crucial in supporting the cultivation of critical thinking in the context of collaborative problem-solving. These are the precise outcomes, as follows:

Various learning stages influenced critical thinking positively, without significant intergroup differences (chi 2  = 3.15, P  = 0.21 > 0.05). High school was first on the list of effect sizes (ES = 1.36, P  < 0.01), then higher education (ES = 0.78, P  < 0.01), and middle school (ES = 0.73, P  < 0.01). These results show that, despite the learning stage’s beneficial influence on cultivating learners’ critical thinking, we are unable to explain why it is essential for cultivating critical thinking in the context of collaborative problem-solving.

Different teaching types had varying degrees of positive impact on critical thinking, with significant intergroup differences (chi 2  = 7.20, P  < 0.05). The effect size was ranked as follows: mixed courses (ES = 1.34, P  < 0.01), integrated courses (ES = 0.81, P  < 0.01), and independent courses (ES = 0.27, P  < 0.01). These results indicate that the most effective approach to cultivate critical thinking utilizing collaborative problem solving is through the teaching type of mixed courses.

Various intervention durations significantly improved critical thinking, and there were significant intergroup differences (chi 2  = 12.18, P  < 0.01). The effect sizes related to this variable showed a tendency to increase with longer intervention durations. The improvement in critical thinking reached a significant level (ES = 0.85, P  < 0.01) after more than 12 weeks of training. These findings indicate that the intervention duration and critical thinking’s impact are positively correlated, with a longer intervention duration having a greater effect.

Different learning scaffolds influenced critical thinking positively, with significant intergroup differences (chi 2  = 9.03, P  < 0.01). The resource-supported learning scaffold (ES = 0.69, P  < 0.01) acquired a medium-to-higher level of impact, the technique-supported learning scaffold (ES = 0.63, P  < 0.01) also attained a medium-to-higher level of impact, and the teacher-supported learning scaffold (ES = 0.92, P  < 0.01) displayed a high level of significant impact. These results show that the learning scaffold with teacher support has the greatest impact on cultivating critical thinking.

Various group sizes influenced critical thinking positively, and the intergroup differences were statistically significant (chi 2  = 8.77, P  < 0.05). Critical thinking showed a general declining trend with increasing group size. The overall effect size of 2–3 people in this situation was the biggest (ES = 0.99, P  < 0.01), and when the group size was greater than 7 people, the improvement in critical thinking was at the lower-middle level (ES < 0.5, P  < 0.01). These results show that the impact on critical thinking is positively connected with group size, and as group size grows, so does the overall impact.

Various measuring tools influenced critical thinking positively, with significant intergroup differences (chi 2  = 0.08, P  = 0.78 > 0.05). In this situation, the self-adapting measurement tools obtained an upper-medium level of effect (ES = 0.78), whereas the complete effect size of the standardized measurement tools was the largest, achieving a significant level of effect (ES = 0.84, P  < 0.01). These results show that, despite the beneficial influence of the measuring tool on cultivating critical thinking, we are unable to explain why it is crucial in fostering the growth of critical thinking by utilizing the approach of collaborative problem-solving.

Different subject areas had a greater impact on critical thinking, and the intergroup differences were statistically significant (chi 2  = 13.36, P  < 0.05). Mathematics had the greatest overall impact, achieving a significant level of effect (ES = 1.68, P  < 0.01), followed by science (ES = 1.25, P  < 0.01) and medical science (ES = 0.87, P  < 0.01), both of which also achieved a significant level of effect. Programming technology was the least effective (ES = 0.39, P  < 0.01), only having a medium-low degree of effect compared to education (ES = 0.72, P  < 0.01) and other fields (such as language, art, and social sciences) (ES = 0.58, P  < 0.01). These results suggest that scientific fields (e.g., mathematics, science) may be the most effective subject areas for cultivating critical thinking utilizing the approach of collaborative problem-solving.

The effectiveness of collaborative problem solving with regard to teaching critical thinking

According to this meta-analysis, using collaborative problem-solving as an intervention strategy in critical thinking teaching has a considerable amount of impact on cultivating learners’ critical thinking as a whole and has a favorable promotional effect on the two dimensions of critical thinking. According to certain studies, collaborative problem solving, the most frequently used critical thinking teaching strategy in curriculum instruction can considerably enhance students’ critical thinking (e.g., Liang et al., 2017 ; Liu et al., 2020 ; Cindy, 2004 ). This meta-analysis provides convergent data support for the above research views. Thus, the findings of this meta-analysis not only effectively address the first research query regarding the overall effect of cultivating critical thinking and its impact on the two dimensions of critical thinking (i.e., attitudinal tendency and cognitive skills) utilizing the approach of collaborative problem-solving, but also enhance our confidence in cultivating critical thinking by using collaborative problem-solving intervention approach in the context of classroom teaching.

Furthermore, the associated improvements in attitudinal tendency are much stronger, but the corresponding improvements in cognitive skill are only marginally better. According to certain studies, cognitive skill differs from the attitudinal tendency in classroom instruction; the cultivation and development of the former as a key ability is a process of gradual accumulation, while the latter as an attitude is affected by the context of the teaching situation (e.g., a novel and exciting teaching approach, challenging and rewarding tasks) (Halpern, 2001 ; Wei and Hong, 2022 ). Collaborative problem-solving as a teaching approach is exciting and interesting, as well as rewarding and challenging; because it takes the learners as the focus and examines problems with poor structure in real situations, and it can inspire students to fully realize their potential for problem-solving, which will significantly improve their attitudinal tendency toward solving problems (Liu et al., 2020 ). Similar to how collaborative problem-solving influences attitudinal tendency, attitudinal tendency impacts cognitive skill when attempting to solve a problem (Liu et al., 2020 ; Zhang et al., 2022 ), and stronger attitudinal tendencies are associated with improved learning achievement and cognitive ability in students (Sison, 2008 ; Zhang et al., 2022 ). It can be seen that the two specific dimensions of critical thinking as well as critical thinking as a whole are affected by collaborative problem-solving, and this study illuminates the nuanced links between cognitive skills and attitudinal tendencies with regard to these two dimensions of critical thinking. To fully develop students’ capacity for critical thinking, future empirical research should pay closer attention to cognitive skills.

The moderating effects of collaborative problem solving with regard to teaching critical thinking

In order to further explore the key factors that influence critical thinking, exploring possible moderating effects that might produce considerable heterogeneity was done using subgroup analysis. The findings show that the moderating factors, such as the teaching type, learning stage, group size, learning scaffold, duration of the intervention, measuring tool, and the subject area included in the 36 experimental designs, could all support the cultivation of collaborative problem-solving in critical thinking. Among them, the effect size differences between the learning stage and measuring tool are not significant, which does not explain why these two factors are crucial in supporting the cultivation of critical thinking utilizing the approach of collaborative problem-solving.

In terms of the learning stage, various learning stages influenced critical thinking positively without significant intergroup differences, indicating that we are unable to explain why it is crucial in fostering the growth of critical thinking.

Although high education accounts for 70.89% of all empirical studies performed by researchers, high school may be the appropriate learning stage to foster students’ critical thinking by utilizing the approach of collaborative problem-solving since it has the largest overall effect size. This phenomenon may be related to student’s cognitive development, which needs to be further studied in follow-up research.

With regard to teaching type, mixed course teaching may be the best teaching method to cultivate students’ critical thinking. Relevant studies have shown that in the actual teaching process if students are trained in thinking methods alone, the methods they learn are isolated and divorced from subject knowledge, which is not conducive to their transfer of thinking methods; therefore, if students’ thinking is trained only in subject teaching without systematic method training, it is challenging to apply to real-world circumstances (Ruggiero, 2012 ; Hu and Liu, 2015 ). Teaching critical thinking as mixed course teaching in parallel to other subject teachings can achieve the best effect on learners’ critical thinking, and explicit critical thinking instruction is more effective than less explicit critical thinking instruction (Bensley and Spero, 2014 ).

In terms of the intervention duration, with longer intervention times, the overall effect size shows an upward tendency. Thus, the intervention duration and critical thinking’s impact are positively correlated. Critical thinking, as a key competency for students in the 21st century, is difficult to get a meaningful improvement in a brief intervention duration. Instead, it could be developed over a lengthy period of time through consistent teaching and the progressive accumulation of knowledge (Halpern, 2001 ; Hu and Liu, 2015 ). Therefore, future empirical studies ought to take these restrictions into account throughout a longer period of critical thinking instruction.

With regard to group size, a group size of 2–3 persons has the highest effect size, and the comprehensive effect size decreases with increasing group size in general. This outcome is in line with some research findings; as an example, a group composed of two to four members is most appropriate for collaborative learning (Schellens and Valcke, 2006 ). However, the meta-analysis results also indicate that once the group size exceeds 7 people, small groups cannot produce better interaction and performance than large groups. This may be because the learning scaffolds of technique support, resource support, and teacher support improve the frequency and effectiveness of interaction among group members, and a collaborative group with more members may increase the diversity of views, which is helpful to cultivate critical thinking utilizing the approach of collaborative problem-solving.

With regard to the learning scaffold, the three different kinds of learning scaffolds can all enhance critical thinking. Among them, the teacher-supported learning scaffold has the largest overall effect size, demonstrating the interdependence of effective learning scaffolds and collaborative problem-solving. This outcome is in line with some research findings; as an example, a successful strategy is to encourage learners to collaborate, come up with solutions, and develop critical thinking skills by using learning scaffolds (Reiser, 2004 ; Xu et al., 2022 ); learning scaffolds can lower task complexity and unpleasant feelings while also enticing students to engage in learning activities (Wood et al., 2006 ); learning scaffolds are designed to assist students in using learning approaches more successfully to adapt the collaborative problem-solving process, and the teacher-supported learning scaffolds have the greatest influence on critical thinking in this process because they are more targeted, informative, and timely (Xu et al., 2022 ).

With respect to the measuring tool, despite the fact that standardized measurement tools (such as the WGCTA, CCTT, and CCTST) have been acknowledged as trustworthy and effective by worldwide experts, only 54.43% of the research included in this meta-analysis adopted them for assessment, and the results indicated no intergroup differences. These results suggest that not all teaching circumstances are appropriate for measuring critical thinking using standardized measurement tools. “The measuring tools for measuring thinking ability have limits in assessing learners in educational situations and should be adapted appropriately to accurately assess the changes in learners’ critical thinking.”, according to Simpson and Courtney ( 2002 , p. 91). As a result, in order to more fully and precisely gauge how learners’ critical thinking has evolved, we must properly modify standardized measuring tools based on collaborative problem-solving learning contexts.

With regard to the subject area, the comprehensive effect size of science departments (e.g., mathematics, science, medical science) is larger than that of language arts and social sciences. Some recent international education reforms have noted that critical thinking is a basic part of scientific literacy. Students with scientific literacy can prove the rationality of their judgment according to accurate evidence and reasonable standards when they face challenges or poorly structured problems (Kyndt et al., 2013 ), which makes critical thinking crucial for developing scientific understanding and applying this understanding to practical problem solving for problems related to science, technology, and society (Yore et al., 2007 ).

Suggestions for critical thinking teaching

Other than those stated in the discussion above, the following suggestions are offered for critical thinking instruction utilizing the approach of collaborative problem-solving.

First, teachers should put a special emphasis on the two core elements, which are collaboration and problem-solving, to design real problems based on collaborative situations. This meta-analysis provides evidence to support the view that collaborative problem-solving has a strong synergistic effect on promoting students’ critical thinking. Asking questions about real situations and allowing learners to take part in critical discussions on real problems during class instruction are key ways to teach critical thinking rather than simply reading speculative articles without practice (Mulnix, 2012 ). Furthermore, the improvement of students’ critical thinking is realized through cognitive conflict with other learners in the problem situation (Yang et al., 2008 ). Consequently, it is essential for teachers to put a special emphasis on the two core elements, which are collaboration and problem-solving, and design real problems and encourage students to discuss, negotiate, and argue based on collaborative problem-solving situations.

Second, teachers should design and implement mixed courses to cultivate learners’ critical thinking, utilizing the approach of collaborative problem-solving. Critical thinking can be taught through curriculum instruction (Kuncel, 2011 ; Leng and Lu, 2020 ), with the goal of cultivating learners’ critical thinking for flexible transfer and application in real problem-solving situations. This meta-analysis shows that mixed course teaching has a highly substantial impact on the cultivation and promotion of learners’ critical thinking. Therefore, teachers should design and implement mixed course teaching with real collaborative problem-solving situations in combination with the knowledge content of specific disciplines in conventional teaching, teach methods and strategies of critical thinking based on poorly structured problems to help students master critical thinking, and provide practical activities in which students can interact with each other to develop knowledge construction and critical thinking utilizing the approach of collaborative problem-solving.

Third, teachers should be more trained in critical thinking, particularly preservice teachers, and they also should be conscious of the ways in which teachers’ support for learning scaffolds can promote critical thinking. The learning scaffold supported by teachers had the greatest impact on learners’ critical thinking, in addition to being more directive, targeted, and timely (Wood et al., 2006 ). Critical thinking can only be effectively taught when teachers recognize the significance of critical thinking for students’ growth and use the proper approaches while designing instructional activities (Forawi, 2016 ). Therefore, with the intention of enabling teachers to create learning scaffolds to cultivate learners’ critical thinking utilizing the approach of collaborative problem solving, it is essential to concentrate on the teacher-supported learning scaffolds and enhance the instruction for teaching critical thinking to teachers, especially preservice teachers.

Implications and limitations

There are certain limitations in this meta-analysis, but future research can correct them. First, the search languages were restricted to English and Chinese, so it is possible that pertinent studies that were written in other languages were overlooked, resulting in an inadequate number of articles for review. Second, these data provided by the included studies are partially missing, such as whether teachers were trained in the theory and practice of critical thinking, the average age and gender of learners, and the differences in critical thinking among learners of various ages and genders. Third, as is typical for review articles, more studies were released while this meta-analysis was being done; therefore, it had a time limit. With the development of relevant research, future studies focusing on these issues are highly relevant and needed.

Conclusions

The subject of the magnitude of collaborative problem-solving’s impact on fostering students’ critical thinking, which received scant attention from other studies, was successfully addressed by this study. The question of the effectiveness of collaborative problem-solving in promoting students’ critical thinking was addressed in this study, which addressed a topic that had gotten little attention in earlier research. The following conclusions can be made:

Regarding the results obtained, collaborative problem solving is an effective teaching approach to foster learners’ critical thinking, with a significant overall effect size (ES = 0.82, z  = 12.78, P  < 0.01, 95% CI [0.69, 0.95]). With respect to the dimensions of critical thinking, collaborative problem-solving can significantly and effectively improve students’ attitudinal tendency, and the comprehensive effect is significant (ES = 1.17, z  = 7.62, P  < 0.01, 95% CI [0.87, 1.47]); nevertheless, it falls short in terms of improving students’ cognitive skills, having only an upper-middle impact (ES = 0.70, z  = 11.55, P  < 0.01, 95% CI [0.58, 0.82]).

As demonstrated by both the results and the discussion, there are varying degrees of beneficial effects on students’ critical thinking from all seven moderating factors, which were found across 36 studies. In this context, the teaching type (chi 2  = 7.20, P  < 0.05), intervention duration (chi 2  = 12.18, P  < 0.01), subject area (chi 2  = 13.36, P  < 0.05), group size (chi 2  = 8.77, P  < 0.05), and learning scaffold (chi 2  = 9.03, P  < 0.01) all have a positive impact on critical thinking, and they can be viewed as important moderating factors that affect how critical thinking develops. Since the learning stage (chi 2  = 3.15, P  = 0.21 > 0.05) and measuring tools (chi 2  = 0.08, P  = 0.78 > 0.05) did not demonstrate any significant intergroup differences, we are unable to explain why these two factors are crucial in supporting the cultivation of critical thinking in the context of collaborative problem-solving.

Data availability

All data generated or analyzed during this study are included within the article and its supplementary information files, and the supplementary information files are available in the Dataverse repository: https://doi.org/10.7910/DVN/IPFJO6 .

Bensley DA, Spero RA (2014) Improving critical thinking skills and meta-cognitive monitoring through direct infusion. Think Skills Creat 12:55–68. https://doi.org/10.1016/j.tsc.2014.02.001

Article   Google Scholar  

Castle A (2009) Defining and assessing critical thinking skills for student radiographers. Radiography 15(1):70–76. https://doi.org/10.1016/j.radi.2007.10.007

Chen XD (2013) An empirical study on the influence of PBL teaching model on critical thinking ability of non-English majors. J PLA Foreign Lang College 36 (04):68–72

Google Scholar  

Cohen A (1992) Antecedents of organizational commitment across occupational groups: a meta-analysis. J Organ Behav. https://doi.org/10.1002/job.4030130602

Cooper H (2010) Research synthesis and meta-analysis: a step-by-step approach, 4th edn. Sage, London, England

Cindy HS (2004) Problem-based learning: what and how do students learn? Educ Psychol Rev 51(1):31–39

Duch BJ, Gron SD, Allen DE (2001) The power of problem-based learning: a practical “how to” for teaching undergraduate courses in any discipline. Stylus Educ Sci 2:190–198

Ennis RH (1989) Critical thinking and subject specificity: clarification and needed research. Educ Res 18(3):4–10. https://doi.org/10.3102/0013189x018003004

Facione PA (1990) Critical thinking: a statement of expert consensus for purposes of educational assessment and instruction. Research findings and recommendations. Eric document reproduction service. https://eric.ed.gov/?id=ed315423

Facione PA, Facione NC (1992) The California Critical Thinking Dispositions Inventory (CCTDI) and the CCTDI test manual. California Academic Press, Millbrae, CA

Forawi SA (2016) Standard-based science education and critical thinking. Think Skills Creat 20:52–62. https://doi.org/10.1016/j.tsc.2016.02.005

Halpern DF (2001) Assessing the effectiveness of critical thinking instruction. J Gen Educ 50(4):270–286. https://doi.org/10.2307/27797889

Hu WP, Liu J (2015) Cultivation of pupils’ thinking ability: a five-year follow-up study. Psychol Behav Res 13(05):648–654. https://doi.org/10.3969/j.issn.1672-0628.2015.05.010

Huber K (2016) Does college teach critical thinking? A meta-analysis. Rev Educ Res 86(2):431–468. https://doi.org/10.3102/0034654315605917

Kek MYCA, Huijser H (2011) The power of problem-based learning in developing critical thinking skills: preparing students for tomorrow’s digital futures in today’s classrooms. High Educ Res Dev 30(3):329–341. https://doi.org/10.1080/07294360.2010.501074

Kuncel NR (2011) Measurement and meaning of critical thinking (Research report for the NRC 21st Century Skills Workshop). National Research Council, Washington, DC

Kyndt E, Raes E, Lismont B, Timmers F, Cascallar E, Dochy F (2013) A meta-analysis of the effects of face-to-face cooperative learning. Do recent studies falsify or verify earlier findings? Educ Res Rev 10(2):133–149. https://doi.org/10.1016/j.edurev.2013.02.002

Leng J, Lu XX (2020) Is critical thinking really teachable?—A meta-analysis based on 79 experimental or quasi experimental studies. Open Educ Res 26(06):110–118. https://doi.org/10.13966/j.cnki.kfjyyj.2020.06.011

Liang YZ, Zhu K, Zhao CL (2017) An empirical study on the depth of interaction promoted by collaborative problem solving learning activities. J E-educ Res 38(10):87–92. https://doi.org/10.13811/j.cnki.eer.2017.10.014

Lipsey M, Wilson D (2001) Practical meta-analysis. International Educational and Professional, London, pp. 92–160

Liu Z, Wu W, Jiang Q (2020) A study on the influence of problem based learning on college students’ critical thinking-based on a meta-analysis of 31 studies. Explor High Educ 03:43–49

Morris SB (2008) Estimating effect sizes from pretest-posttest-control group designs. Organ Res Methods 11(2):364–386. https://doi.org/10.1177/1094428106291059

Article   ADS   Google Scholar  

Mulnix JW (2012) Thinking critically about critical thinking. Educ Philos Theory 44(5):464–479. https://doi.org/10.1111/j.1469-5812.2010.00673.x

Naber J, Wyatt TH (2014) The effect of reflective writing interventions on the critical thinking skills and dispositions of baccalaureate nursing students. Nurse Educ Today 34(1):67–72. https://doi.org/10.1016/j.nedt.2013.04.002

National Research Council (2012) Education for life and work: developing transferable knowledge and skills in the 21st century. The National Academies Press, Washington, DC

Niu L, Behar HLS, Garvan CW (2013) Do instructional interventions influence college students’ critical thinking skills? A meta-analysis. Educ Res Rev 9(12):114–128. https://doi.org/10.1016/j.edurev.2012.12.002

Peng ZM, Deng L (2017) Towards the core of education reform: cultivating critical thinking skills as the core of skills in the 21st century. Res Educ Dev 24:57–63. https://doi.org/10.14121/j.cnki.1008-3855.2017.24.011

Reiser BJ (2004) Scaffolding complex learning: the mechanisms of structuring and problematizing student work. J Learn Sci 13(3):273–304. https://doi.org/10.1207/s15327809jls1303_2

Ruggiero VR (2012) The art of thinking: a guide to critical and creative thought, 4th edn. Harper Collins College Publishers, New York

Schellens T, Valcke M (2006) Fostering knowledge construction in university students through asynchronous discussion groups. Comput Educ 46(4):349–370. https://doi.org/10.1016/j.compedu.2004.07.010

Sendag S, Odabasi HF (2009) Effects of an online problem based learning course on content knowledge acquisition and critical thinking skills. Comput Educ 53(1):132–141. https://doi.org/10.1016/j.compedu.2009.01.008

Sison R (2008) Investigating Pair Programming in a Software Engineering Course in an Asian Setting. 2008 15th Asia-Pacific Software Engineering Conference, pp. 325–331. https://doi.org/10.1109/APSEC.2008.61

Simpson E, Courtney M (2002) Critical thinking in nursing education: literature review. Mary Courtney 8(2):89–98

Stewart L, Tierney J, Burdett S (2006) Do systematic reviews based on individual patient data offer a means of circumventing biases associated with trial publications? Publication bias in meta-analysis. John Wiley and Sons Inc, New York, pp. 261–286

Tiwari A, Lai P, So M, Yuen K (2010) A comparison of the effects of problem-based learning and lecturing on the development of students’ critical thinking. Med Educ 40(6):547–554. https://doi.org/10.1111/j.1365-2929.2006.02481.x

Wood D, Bruner JS, Ross G (2006) The role of tutoring in problem solving. J Child Psychol Psychiatry 17(2):89–100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x

Wei T, Hong S (2022) The meaning and realization of teachable critical thinking. Educ Theory Practice 10:51–57

Xu EW, Wang W, Wang QX (2022) A meta-analysis of the effectiveness of programming teaching in promoting K-12 students’ computational thinking. Educ Inf Technol. https://doi.org/10.1007/s10639-022-11445-2

Yang YC, Newby T, Bill R (2008) Facilitating interactions through structured web-based bulletin boards: a quasi-experimental study on promoting learners’ critical thinking skills. Comput Educ 50(4):1572–1585. https://doi.org/10.1016/j.compedu.2007.04.006

Yore LD, Pimm D, Tuan HL (2007) The literacy component of mathematical and scientific literacy. Int J Sci Math Educ 5(4):559–589. https://doi.org/10.1007/s10763-007-9089-4

Zhang T, Zhang S, Gao QQ, Wang JH (2022) Research on the development of learners’ critical thinking in online peer review. Audio Visual Educ Res 6:53–60. https://doi.org/10.13811/j.cnki.eer.2022.06.08

Download references

Acknowledgements

This research was supported by the graduate scientific research and innovation project of Xinjiang Uygur Autonomous Region named “Research on in-depth learning of high school information technology courses for the cultivation of computing thinking” (No. XJ2022G190) and the independent innovation fund project for doctoral students of the College of Educational Science of Xinjiang Normal University named “Research on project-based teaching of high school information technology courses from the perspective of discipline core literacy” (No. XJNUJKYA2003).

Author information

Authors and affiliations.

College of Educational Science, Xinjiang Normal University, 830017, Urumqi, Xinjiang, China

Enwei Xu, Wei Wang & Qingxia Wang

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Enwei Xu or Wei Wang .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

Additional information.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary tables, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Xu, E., Wang, W. & Wang, Q. The effectiveness of collaborative problem solving in promoting students’ critical thinking: A meta-analysis based on empirical literature. Humanit Soc Sci Commun 10 , 16 (2023). https://doi.org/10.1057/s41599-023-01508-1

Download citation

Received : 07 August 2022

Accepted : 04 January 2023

Published : 11 January 2023

DOI : https://doi.org/10.1057/s41599-023-01508-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Impacts of online collaborative learning on students’ intercultural communication apprehension and intercultural communicative competence.

• Hoa Thi Hoang Chau
• Hung Phu Bui
• Quynh Thi Huong Dinh

Education and Information Technologies (2024)

Exploring the effects of digital technology on deep learning: a meta-analysis

The impacts of computer-supported collaborative learning on students’ critical thinking: a meta-analysis.

• Yoseph Gebrehiwot Tedla
• Hsiu-Ling Chen

Sustainable electricity generation and farm-grid utilization from photovoltaic aquaculture: a bibliometric analysis

• A. A. Amusa
• M. Alhassan

International Journal of Environmental Science and Technology (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

BRIEF RESEARCH REPORT article

Effects of online problem-solving instruction and identification attitude toward instructional strategies on students' creativity.

• College of International Relations, Huaqiao University, Xiamen, China

Problem-solving ability is an essential part of daily life. Thus, curiosity and a thirst for knowledge should be cultivated in students to help them develop problem solving and independent thinking skills. Along with positive attitudes and an active disposition, these abilities are needed to solve problems throughout the lifespan and develop -confidence. To achieve educational objectives in the context of globalization, creative ability is necessary for generating competitive advantages. Therefore, creative thinking, critical thinking, and problem-solving ability are important basic competencies needed for future world citizens. Creativity should also be integrated into subject teaching to cultivate students' lifelong learning and a creative attitude toward life. A questionnaire was distributed to 420 students in colleges and universities in Fujian, China. After removing invalid and incomplete responses, 363 copies were found to be valid yielding a response rate of 86%. Findings indicate that the new generation requires high levels of support to develop creativity and integrate diverse subjects such as nature, humanities, and technology. A rich imagination is needed to root creativity in the new generation.

Introduction

Problem solving is ubiquitous in modern life and an essential skill for overcoming the problems we encounter daily. Problems can be overcome using problem-solving principles and creative inspiration from individuals ( Hao et al., 2016 ). Thus, students' curiosity and thirst for knowledge should be cultivated to develop their problem solving and independent thinking abilities. An active approach and positive attitude to solving problems may enhance self-confidence and the ability to cope with challenges.

Education aims to cultivate healthy personalities, thinking, judgment, and creativity ( Su et al., 2014 ). Essentially, education is the learning process to expand students' potential and cultivate their ability to adapt to—and improve—their environment. Basic goals of education should include self-expression, independent thinking, active inquiry, and problem solving. The curriculum goals should be life-centered to develop individuals' potential, cultivate scientific knowledge and skills, and help students adapt to the demands of modern life ( Atmatzidou et al., 2018 ). Education aims to deliver basic knowledge, cultivate physical and mental development, inquiry, and reflection, and create healthy citizens through activities involving interaction between individuals, individuals and society, and society and nature. To achieve educational objectives, students should be guided to develop their performance and creation abilities, research and active exploration abilities, independent thinking and problem-solving abilities. In the current globalization context, creative abilities are required for building competitive advantages. Accordingly, creative thinking, critical thinking, and problem-solving abilities are key skills for future world citizens. The cultivation of creativity should also be integrated into subject instruction, so that students develop their lifelong learning and creative attitudes toward life. Many countries are eager to cultivate creative new generations and promote the development of local business and humanistic technological education. It has become a national platform for the new generations of international technological art ( Zhang and Chu, 2016 ). In particular, the traditional productivity-oriented competition model is slowly being transformed to creativity-oriented industries. Innovation capability is likely to bring competitive advantages in the Internet information age. As the field of information technology grows exponentially, innovation capability has become more important. However, if opportunities for development are missed, it can be difficult to catch up as the need for creativity is likely to grow in the foreseeable future.

In this study we focus on student creativity and how it is affected by online problem-solving instruction and identification of attitudes toward instructional strategies. Our purpose is to help the new generation develop creativity and a rich imagination to integrate the power of nature, humanities, and technology.

Literature Review and Hypothesis

Su et al. (2017) proposed that teachers who use effective instructional strategies allow students to successfully negotiate the challenges of life, as effective instructional strategies may enhance students' problem-solving ability. Deeper relationships between teachers and students also result in better learning motivation for students. Art-related activities were used to observe the factors affecting preschool children's problem-solving ability ( Calvo et al., 2018 ). These factors included the cognition of problem goals, the development of perception ability, individual experience, interaction among peers, and resource assistance provided by teachers' instructional strategies. ( LaForce et al., 2017 ) pointed out that identifying problem-related data is an essential step in the problem-solving process, i.e., the process of acquiring data, judging data, reducing data coverage, or linking relevant data ( Wu et al., 2020a ). Teachers' instructional strategies for online problem solving also affect student performance. The following hypothesis was therefore established for this study.

H1: Online problem-solving instruction has a significant positive correlation with identification attitude.

Lu et al. (2017) consider that teachers can enhance students' problem-solving ability and cultivate their problem-finding skills through instructional strategies guiding discussion of current affairs. Instructional strategies and the use of multimedia in technology education can induce students' identification attitudes and learning motivation, ultimately enhancing learning effectiveness and facilitating the development of imagination and creativity. Students with identification attitudes toward strategies could design problem-solving methods using science ( Newhouse, 2017 ). The students understood that innovation was not necessarily the novel creation of “something from nothing” but might involve modification and new development based on existing affairs ( Wu et al., 2020b ). Achilleos et al. (2019) regard attitude toward education instructional strategies as the most important factor in students' creativity learning, where teachers, social and cultural factors, and experience in learning a foreign language revealed significant correlations. Our second hypothesis was therefore presented for this study.

H2: Identification attitude shows strong positive correlations with creativity.

Hsieh et al. (2017) posit that science-related thinking, discovery, and creation can be regarded as the research component of problem solving. Creativity is characterized by keenness, fluency, flexibility, originality, and elaboration—a kind of mental intelligence to generate distinct new concepts from known experiences or knowledge to solve problems with creative methods. Creativity can also be the application of known information, based on targeted outcomes, to generate novel, unique, and valuable new concepts or a new product or technology, unexplored innovative concepts or problem-solving abilities ( Wu et al., 2021 ). Joachim et al. (2018) consider creativity as a part of problem solving, as problem-solving characteristics often involve novel thinking, strong motivation and determination to present the important status of the solution in the latent process of problem solving. However, Joachim et al.'s (2018) views on creativity and problem-solving have largely been unexplored to date. Novel performance at any level of the creative process could be considered as creation. Rietz et al. (2019) stated that life brings diverse problems and the key to addressing these lies in creativity. Only when people invest more attention in creativity can problems be solved leading to optimum solutions for life's challenges. This gives rise to our third hypothesis.

H3: Online problem-solving instruction reveals strong positive correlations with creativity

Methodology

Operational definitions, online problem-solving instruction.

Referring to Chen et al. (2019) , the dimensions of online problem-solving instruction in this study were as follows.

1. Exercise example: Examples to illustrate teaching goals are provided as part of teachers' instruction. Students can learn effective problem-solving skills by observing experts' problem-solving interpretation and demonstration step-by-step.

2. Problem orientation: Problem-oriented learning refers to teachers giving carefully-designed situational problems to students, who start from a problem and proceed to problem solving and learning. After self-learning, students participate in team discussion or discussions with teachers. With constant trials, solutions are eventually proposed.

Identification Attitude

The dimensions for identification attitude toward learning are based on Tang et al. (2019) and contain the following three components.

1. Cognitive component: This refers to an individual's belief in or knowledge of specific matters. The cognition of attitude refers to evaluation of meaning from factual statements presented, i.e., an individual may form an attitude for or against a particular object. For instance, students understand that teachers have rich professional knowledge and can present materials with good organization.

2. Affective component: The affective or emotional component refers to an individual's emotions and feelings, including positive and negative feelings of respect and contempt, like and dislike, sympathy and exclusion. For example, students evaluating a teacher as a friendly person would have positive feelings about the teacher and want to develop that relationship.

3. Behavioral component: Behavior refers to an individual's response tendency to attitude objects, i.e., an individual's explicit behavioral performance when acting in relation to objects. Possible responses include approach, avoidance, or indifference. For instance, students might accept their teachers' arrangement of an activity with respect and actively ask teachers questions.

Kim et al. (2019) consider creativity includes basic cognitive abilities of divergent thinking, and that such abilities can be understood through testing tools or observation.

1. Fluency: Fluency refers to the quantity of a person's concept output, i.e., the ability to generate possible programs or solutions. A student with fluent thinking would propose several responses at the concept generation stage.

2. Flexibility: Flexibility is the ability to change thinking direction, i.e., being able to think of different methods when problems occur, to find out distinct applications or new concepts.

3. Originality: Originality refers to generation of unique and novel ideas, i.e., doing unexpected things or having the ability to see others' points of view.

4. Elaboration: Elaboration is a supplementary idea that refers to the ability to add new ideas to an original concept, i.e., the ability to increase novel concepts or build on existing ideas or basic concepts.

Research Objective

There are 89 colleges and universities in Fujian, China (50 colleges and 39 universities). Students in these institutions in Fujian comprised the research sample, and we distributed 420 copies of our questionnaire to them. After removing invalid and incomplete questionnaires, a total of 363 valid copies were returned, with a response rate of 86%.

This research focused on discussing online problems about teaching and teaching strategies. It used experimental design and online problem solving to do experimental research for 2 hours every week for 24 weeks (48 hours in total). To analyze data from the questionnaire, Structural Equation Modeling (SEM) was used. We followed a two-stage analysis of goodness-of-fit and model verification. Confirmatory Factor Analysis (CFA) was first executed, aiming to test complex variables in the model by deleting measured variables with negative effects on the cause-and-effect analysis. We then proceeded with path analysis with the modified model. Path analysis aims to estimate the path relationship among variables. Without testing complex variables through CFA, the path analysis might be affected by complex variables resulting in poor goodness-of-fit or an insignificant model path. Amos 18.0 was used in this study for the model fit test. The measurement result of CMIN/DF is considered good if lower than five and excellent if lower than three; Goodness-of-Fit Index (GFI), Adjusted Goodness-of-Fit index (AGFI), Normed Fit Index (NFI), Incremental Fit Index (IFI), Tucker-Lewis Index (TLI), and Comparative Fit Index (CFI) are considered good if higher than 0.9; and Root Mean Square Residual (RMR), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean Square Residual (SRMR) are good if values are lower than 0.05.

Factor Analysis

Two factors of “exercise example” (eigenvalue = 4.638, α = 0.88) and “problem orientation” (eigenvalue = 3.751, α = 0.85) were extracted from the scale of instructional strategies for online problem solving. The cumulative covariance accounted for was 72.683%. Three factors were extracted from the identification attitude scale: “cognitive component” (eigenvalue = 2.463, α = 0.81), “affective component” (eigenvalue = 1.754, α = 0.83), and “behavioral component” (eigenvalue = 1.491, α = 0.84). The cumulative covariance reached 73.942%. Four factors were extracted from the creativity scale: “fluency” (eigenvalue = 2.461, α = 0.84), “flexibility” (eigenvalue = 2.055, α = 0.82), “originality” (eigenvalue = 1.976, α = 0.87), and “elaboration” (eigenvalue = 1.689, α = 0.86). The cumulative covariance accounted for was 79.317%.

Empirical Analysis of SEM

CFA results indicated the convergent and discriminant validity of the model were first observed, with convergent validity describing the reliability of individually observed variables, construct reliability (CR), and average variances extracted (AVE). Values of more than 0.5 indicate good reliability of individually observed variables. The factor loadings of the observed variables in the empirical analysis model were higher than the suggested value. CR should exceed 0.6, although some researchers suggest that 0.5 or above is acceptable. The model calibration results reveal CR was higher than 0.6, and AVE higher than 0.5, thus conforming to the suggested values.

Regarding the calibration results of structural equations, χ 2 / df , RMSEA, GFI, AGFI, RMR, and NFI were also calculated. For χ 2 / df a standard ≦5 is suggested and χ 2 / df = 2.422 ≦ 5 in this study. The standard for RMSEA is ≦0.08; reported here as 0.044 ≦ 0.08. GFI has a suggested standard of ≧0.9 and here it is reported as 0.951 ≧ 0.9. AGFI's suggested standard ≧0.9; it shows AGFI = 0.927 ≧ 0.9 in this study. RMR has a suggested standard of ≦0.05, and here it was reported as 0.023 ≦ 0.05. The NFI standard is ≧0.9; here it presents NFI = 0.937 ≧ 0.9 in this study. The overall model fit is good. The parameter calibration of the structural equation is shown in Table 1 and Figure 1 . The research results reveal instructional strategies for online problem solving → identification attitude: 0.346 *** , that is, H1 was supported. Identification attitude → creativity: 0.375 *** , that is, H2 was supported, and instructional strategies for online problem solving → creativity: 0.425 *** , that is, H3 was supported.

Table 1 . Structural equation modeling result.

Figure 1 . FigureModel path diagram.

The results show that online instructional strategies for online problem solving can enhance students' creativity. Apparently, an expository teaching style is no longer sufficient to cope with challenges encountered. Rather, teachers need to be willing to constantly learn and change their teaching behavior to cope with the rapid development of new technology and enhance teaching efficiency. When conveying new knowledge to beginners, the provision of exercise examples may help students establish new schema to benefit the application to similar situations. When lacking relevant schema, beginners may try to solve problems with trial and error. In this case, exercise examples with experts demonstrating problem-solving steps could benefit students' learning performance in the new field. Problems studied in real life may facilitate students' creativity, drawing on their existing knowledge as they use available resources and unconsciously apply existing knowledge to enhance creative ability. The solutions to problems are unpredictable but require the ability to cope with interaction between people in a given culture or society in different situations. As a result, teachers should make decisions with the consideration of situational changes in the teaching site, i.e., students' ability, performance and teaching schedule, rather than generalizing across all situations. We do not suggest limiting creative thinking or defining set times for enhancing students' creative thinking. Instead, factors that influence creative efficiency, creative value, and curriculum schedules should be taken into account. As teachers plan their teaching activities, they should pay particular attention to students' academic performance and the vicarious experience of teachers or peers. Uysal (2014) believed people can develop their mental ability through learning even without any creative invention. When we face any new concept, it is better to keep an open mind. That way we will realize there is still a lot to be created ( Fernández et al., 2018 ). Labusch et al. (2019) said the development of creativity is not only creating positive thoughts but also turning these advantages into something more refined and broader. Teachers need to provide learning opportunities that students can apply in their daily lives leading to a re-evaluation of their identification attitudes toward instructional strategies. In this case, enhancing students' self-efficacy may assist them in overcoming learning challenges and cultivating a more positive learning attitude.

The research results demonstrate that online problem solving supports students to examine their ideas, chase after knowledge and continually improve their learning. They can freely develop their imaginations and make choices without being limited to find tools suitable for self-performance. They can concentrate on details, retain memories, and calmly think of more elaborate problem-solving approaches. Students draw on plans and organization to make significant progress in their thinking depth, novelty, flexibility, unique style, and diversity of function. To cultivate students' habits of brainstorming and thinking, they must become familiar with the general use of contextual information, and flexibility to change approaches and seek answers. Training flexible thinking is essential so that students can cope with problems with ease, propose various options and generate solutions. Lumsdaine and Lumsdaine (1995) let students learn from each other and modify their own thought. This transition could help them to achieve their potential. Solitary and monotonous learning material can no longer attract students' attention. Teachers need to provide a wider variety of materials and free choices without limit. They could also find more suitable tools for teaching. Therefore, Treffinger and Isaksen (1992) no longer provide model answers. They want students to explore and develop without any restriction. This could also amplify their personal experience and bring more options into it. It enhances student's uniqueness, and this needs overall growth, subjectively and objectively. People should never venerate one over the other. We should also learn to make good use of the conditions and things we already have. The same thing could have an entirely different outcome depending on how we use it ( Aşik and Erktin, 2019 ). Consequently, problem-solving instruction could assist in the cultivation of creativity in students' practice ability or cultivation of independent thinking and problem-solving ability. Teachers should attempt to create beneficial educational environments, cultivating students' learning interests, and enhancing their mental development. With accumulated experience, students can then be encouraged to develop more flexible skills, sensitive perception, and active thinking along with the ability to appropriately express these experiences. This would provide comprehensive preparation for enhancing students' creative thinking ability. Instructional strategies for online problem solving heavily emphasize cooperative discussion, brainstorming, and presentation. Tasks focusing on students' favorite novels and other relevant interests are valuable for sustaining long-term attention. Success in learning does not simply rely on rich knowledge and skillful techniques; affective attitudes also play an important part. Such characteristics may encourage students to positively and actively face problems and logically enhance their learning attitudes step-by-step.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by the Ethical Committee of the Huaqiao University. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

Y-PW performed the initial analyses and approved the submitted version of the manuscript.

Conflict of Interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors thank the reviewers for their valuable comments.

Achilleos, A. P., Mettouris, C., Yeratziotis, A., Papadopoulos, G. A., Pllana, S., Huber, F., et al. (2019). SciChallenge: a social media aware platform for contest-based STEM education and motivation of young students. IEEE Trans. Learn. Technol . 12, 98–111. doi: 10.1109/TLT.2018.2810879

CrossRef Full Text | Google Scholar

Aşik, G., and Erktin, E. (2019). Metacognitive experiences: mediating the relationship between metacognitive knowledge and problem solving. Egitim ve Bilim 44, 85–103. doi: 10.15390/EB.2019.7199

Atmatzidou, S., Demetriadis, S., and Nika, P. (2018). How does the degree of guidance support students' metacognitive and problem solving skills in educational robotics? J. Sci. Educ. Technol. 27, 70–85. doi: 10.1007/s10956-017-9709-x

Calvo, I., Cabanes, I., Quesada, J., and Barambones, O. (2018). A multidisciplinary PBL approach for teaching industrial informatics and robotics in engineering. IEEE Trans. Educ . 61, 21–28. doi: 10.1109/TE.2017.2721907

Chen, S.-Y., Lai, C.-F., Lai, Y.-H., and Su Y, S. (2019). Effect of project-based learning on development of students' creative thinking. Int. J. Electr. Eng. Educ . doi: 10.1177/0020720919846808

Fernández, J., Zúñiga, M. E., Rosas, M. V., and Guerrero, R. A. (2018). Experiences in learning problem-solving through computational thinking. J. Comput. Sci. Technol. 18, 136–142. doi: 10.24215/16666038.18.e15

Hao, J., Liu, L., von Davier, A., Kyllonen, P., and Kitchen, C. (2016). “Collaborative problem solving skills vs. collaboration outcomes: findings from statistical analysis and data mining,” in Proceedings of the 9th International Conference on Educational Data Mining (Raleigh, NC), 382–387.

Google Scholar

Hsieh, J. S. C., Huang, Y.-M., and Wu, W.-C. V. (2017). Technological acceptance of LINE in flipped EFL oral training. Comput. Hum. Behav . 70, 178–190. doi: 10.1016/j.chb.2016.12.066

Joachim, V., Spieth, P., and Heidenreich, S. (2018). Active innovation resistance: an empirical study on functional and psychological barriers to innovation adoption in different contexts. Ind. Mark. Manag. 71, 95–107. doi: 10.1016/j.indmarman.2017.12.011

Kim, J., Jordan, S. S., Franklin, C., and Froerer, A. (2019). Is solution-focused brief therapy evidence-based: an update 10 years later. Fam. Soc. 100, 127–138. doi: 10.1177/1044389419841688

Labusch, A., Eickelmann, B., and Vennemann, M. (2019). “Computational thinking processes and their congruence with problem-solving and information processing,” in Proceedings of the Computational Thinking Education , 65–78. doi: 10.1007/978-981-13-6528-7_5

LaForce, M., Noble, E., and Blackwell, C. K. (2017). Problem-based learning (PBL) and student interest in STEM careers: the roles of motivation and ability beliefs. Educ. Sci. 7:92. doi: 10.3390/educsci7040092

Lu, X., Wang, D., and Yu, D. (2017). Effect of solution-focused brief therapy-based on exercise prescription intervention on adolescent mental health. Rev. Argentina de Clin. Psicol . 26, 347–354. doi: 10.24205/03276716.2017.1035

Lumsdaine, E., and Lumsdaine, M. (1995). Creative Problem Solving-Thinking Skills for Changing World . New York, NY: McGRAW-Hill. doi: 10.1109/45.464655

Newhouse, C. (2017). STEM the boredom: engage students in the Australian curriculum using ICT with problem-based learning and assessment. J. Sci. Educ. Technol. 26, 44–57. doi: 10.1007/s10956-016-9650-4

Rietz, T., Benke, I., and Maedche, A. (2019). “The impact of anthropomorphic and functional chatbot design features in enterprise collaboration systems on user acceptance,” in Proceedings of the 14th International Conference Wirtschaftsinformatik (Siegen), 1642-1656.

Su, Y.-S., Ding, T.-J., and Lai, C.-F. (2017). Analysis of students' engagement and learning performance in a social community supported computer programming course. Eurasia J. Math. Sci. Technol. Educ . 13, 6189–6201. doi: 10.12973/eurasia.2017.01058a

Su, Y. S., Yang, J. H., Hwang, W. Y., Huang, S. J., and Tern, M. Y. (2014). Investigating the role of computer-supported annotation in problem solving based teaching: an empirical study of a scratch programming pedagogy. Br. J. Educ. Technol. 45, 647–665. doi: 10.1111/bjet.12058

Tang, K. Y., Hsiao, C. H., and Su, Y. S. (2019). Networking for educational innovations: a bibliometric survey of international publication patterns. Sustainability 11:4608. doi: 10.3390/su11174608

Treffinger, D. J., and Isaksen, S. G. (1992). Creative Problem Solving: An Introduction . Montgomery: Center of Creative Learning, Inc.

Uysal, M. P. (2014). Improving first computer programming experiences: the case of adapting a web-supported and well-structured problem-solving method to a traditional course. Contemp. Educ. Technol . 5, 198–217. doi: 10.30935/cedtech/6125

Wu, T. J., Yuan, K. S., and Yen, D. C. (2021). Leader-member exchange, turnover intention and presenteeism–the moderated mediating effect of perceived organizational support. Curr. Psychol . doi: 10.1007/s12144-021-01825-1

Wu, T. J., Gao, J. Y., Wang, L. Y., and Yuan, K. S. (2020a). Exploring links between polychronicity and job performance from the person–environment fit perspective-the mediating role of well-being. Int. J. Environ. Res. Public Health 17, 3711–3722. doi: 10.3390/ijerph17103711

PubMed Abstract | CrossRef Full Text | Google Scholar

Wu, T. J., Xu, T., Li, L. Q., and Yuan, K. S. (2020b). “Touching with heart, reasoning by truth”! The impact of Brand cues on mini-film advertising effect. Int. J. Advert . 39, 1322–1350. doi: 10.1080/02650487.2020.1755184

Zhang, Y., and Chu, S. K. W. (2016). New ideas on the design of the web-based learning system oriented to problem solving from the perspective of question chain and learning community. Int. Rev. Res. Open Dis. 17, 176–189. doi: 10.19173/irrodl.v17i3.2115

Keywords: online problem, instructional strategies, identification attitude, affective component, creativity

Citation: Wang Y-P (2021) Effects of Online Problem-Solving Instruction and Identification Attitude Toward Instructional Strategies on Students' Creativity. Front. Psychol. 12:771128. doi: 10.3389/fpsyg.2021.771128

Received: 05 September 2021; Accepted: 27 September 2021; Published: 14 October 2021.

Reviewed by:

Copyright © 2021 Wang. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Yi-Ping Wang, 1487774578@qq.com

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Psychol

The Efficacy and Development of Students' Problem-Solving Strategies During Compulsory Schooling: Logfile Analyses

Gyöngyvér molnár.

1 Department of Learning and Instruction, University of Szeged, Szeged, Hungary

Benő Csapó

2 MTA-SZTE Research Group on the Development of Competencies, University of Szeged, Szeged, Hungary

The purpose of this study was to examine the role of exploration strategies students used in the first phase of problem solving. The sample for the study was drawn from 3 rd - to 12 th -grade students (aged 9–18) in Hungarian schools ( n = 4,371). Problems designed in the MicroDYN approach with different levels of complexity were administered to the students via the eDia online platform. Logfile analyses were performed to ascertain the impact of strategy use on the efficacy of problem solving. Students' exploration behavior was coded and clustered through Latent Class Analyses. Several theoretically effective strategies were identified, including the vary-one-thing-at-a-time (VOTAT) strategy and its sub-strategies. The results of the analyses indicate that the use of a theoretically effective strategy, which extract all information required to solve the problem, did not always lead to high performance. Conscious VOTAT strategy users proved to be the best problem solvers followed by non-conscious VOTAT strategy users and non-VOTAT strategy users. In the primary school sub-sample, six qualitatively different strategy class profiles were distinguished. The results shed new light on and provide a new interpretation of previous analyses of the processes involved in complex problem solving. They also highlight the importance of explicit enhancement of problem-solving skills and problem-solving strategies as a tool for knowledge acquisition in new contexts during and beyond school lessons.

Introduction

Computer-based assessment has presented new challenges and opportunities in educational research. A large number of studies have highlighted the importance and advantages of technology-based assessment over traditional paper-based testing (Csapó et al., 2012 ). Three main factors support and motivate the use of technology in educational assessment: (1) the improved efficiency and greater measurement precision in the already established assessment domains (e.g., Csapó et al., 2014 ); (2) the possibility of measuring constructs that would be impossible to measure by other means (e.g., Complex Problem Solving (CPS) 1 ; see Greiff et al., 2012 , 2013 ); and (3) the opportunity of logging and analyzing not only observed variables, but metadata as well (Lotz et al., 2017 ; Tóth et al., 2017 ; Zoanetti and Griffin, 2017 ). Analyzing logfiles may contribute to a deeper and better understanding of the phenomenon under examination. Logfile analyses can provide answers to research questions which could not be answered with traditional assessment techniques.

This study focuses on problem solving, especially on complex problem solving (CPS), which reflects higher-order cognitive processes. Previous research identified three different ways to measure CPS competencies: (1) Microworlds (e.g., Gardner and Berry, 1995 ), (2) formal frameworks (Funke, 2001 , 2010 ) and (3) minimal complex systems (Funke, 2014 ). In this paper, the focus is on the MicroDYN approach, which is a specific form of complex problem solving (CPS) in interactive situations using minimal complex systems (Funke, 2014 ). Recent analyses provide both a new theory and data-based evidence for a global understanding of different problem-solving strategies students employ or could employ in a complex problem-solving environment based on minimal complex systems.

The problem scenarios within the MicroDYN approach consist of a small number of variables and causal relations. From the perspective of the problem solver, solving a MicroDYN problem requires a sequence of continuous activities, in which the outcome of one activity is the input for the next. First, students interact with the simulated system, set values for the input variables, and observe the impacts of these settings on the target (dependent) variable. Then, they plot their conclusion about the causal relationships between the input and output variables on a graph (Phase 1). Next, they manipulate the independent variables again to set their values so that they result in the required values for the target variables (Phase 2).

When it comes to gathering information about a complex problem, as in the MicroDYN scenarios, there may be differences between the exploration strategies in terms of efficacy. Some of them may be more useful for generating knowledge about the system. Tschirgi ( 1980 ) identified different exploration strategies. When control of variables strategies (Greiff et al., 2014 ) were explored, findings showed that the vary-one-thing-at-a-time (VOTAT, Tschirgi, 1980 ; Funke, 2014 ) was the most effective strategy for identifying causal relations between the input and output variables in a minimal complex system (Fischer et al., 2012 ). Participants who employed this strategy tended to acquire more structural knowledge than those who used other strategies (Vollmeyer et al., 1996 ; Kröner et al., 2005 ). With the VOTAT strategy, the problem solver systematically varies only one input variable, while the others remain unchanged. This way, the effect of the variable that has just been changed can be observed directly by monitoring the changes in the output variables. There exist several types of VOTAT strategies.

Using this approach—defining the effectiveness of a strategy on a conceptual level, independently of empirical effectiveness—we developed a labeling system and a mathematical model based on all theoretically effective strategies. Thus, effectiveness was defined and linked to the amount of information extracted. An exploration strategy was defined as theoretically effective if the problem solver was able to extract all the information needed to solve the problem, independently of the application level of the information extracted and of the final achievement. We split the effectiveness of the exploration strategy and the usage and application of the information extracted to be able to solve the problem and control the system with respect to the target values based on the causal knowledge acquired. Systematicity was defined on the level of effectiveness based on the amount of information extracted and on the level of awareness based on the implementation of systematicity in time.

Students' actions were logged and coded according to our input behavior model and then clustered for comparison. We were able to distinguish three different VOTAT strategies and two successful non-VOTAT ones. We empirically tested awareness of the input behavior used in time. Awareness of strategy usage was analyzed by the sequence of the trials used, that is, by the systematicity of the trials used in time. We investigated the effectiveness of and differences in problem-solving behavior between three age groups by conducting latent class analyses to explore and define patterns in qualitatively different VOTAT strategy uses.

Although the assessment of problem solving within the MicroDYN approach is a relatively new area of research, its processes have already been studied in a number of different contexts, including a variety of educational settings with several age groups. Our cross-sectional design allows us to describe differences between age groups and outline the developmental tendencies of input behavior and strategy use among children in the age range covered by our data collection.

Reasoning strategies in complex problem solving

Problem-solving skills have been among the most extensively studied transversal skills over the last decade; they have been investigated in the most prominent comprehensive international large-scale assessments today (e.g., OECD, 2014 ). The common aspects in the different theoretical models are that a problem is characterized by a gap between the current state and the goal state with no immediate solution available (Mayer and Wittrock, 1996 ).

Parallel to the definition of the so-called twenty first-century skills (Griffin et al., 2012 ), recent research on problem solving disregards content knowledge and domain-specific processes. The reason for this is that understanding the structure of unfamiliar problems is more effective when it relies on abstract representation schemas and metacognitive strategies than on specifically relevant example problems (Klahr et al., 2007 ). That is, the focus is more on assessing domain-general problem-solving strategies (Molnár et al., 2017 ), such as complex problem solving, which can be used to solve novel problems, even those arising in interactive situations (Molnár et al., 2013 ).

Logfile analyses make it possible to divide the continuum of a problem-solving process into several scoreable phases by extracting information from the logfile that documents students' problem-solving behavior. In our case, latent class analysis extracts information from the file that logs students' interaction with the simulated system at the beginning of the problem-solving process. The way students manipulate the input (independent) variables represents their reasoning strategy. Log data, on the one hand, make it possible to analyze qualitative differences in these strategies and then their efficiency in terms of how they generate knowledge resulting in the correct plotting of the causal relationship in Phase 1 and then the proper setting to reach the required target value in Phase 2. On the other hand, qualitative strategy data can be quantified, and an alternative scoring system can be devised.

From the perspective of the traditional psychometric approach and method of scoring, these problems form a test task consisting of two scoreable items. The first phase is a knowledge acquisition process, where scores are assigned based on how accurately the causal relationship was plotted. The second phase is knowledge application, where the correctness of the value for the target variable is scored. Such scoring based on two phases of solving MicroDYN problems has been used in a number of previous studies (e.g., Greiff et al., 2013 , 2015 ; Wüstenberg et al., 2014 ; Csapó and Molnár, 2017 ; Greiff and Funke, 2017 ).

To sum up, there is great potential to investigate and cluster the problem-solving behavior and exploration strategy usage of the participants at the beginning of the problem-solving process and correlate the use of a successful exploration strategy with the model-building solution (achievement in Phase 1) observed directly in these simulated problem scenarios. Using logfile analyses (Greiff et al., 2015 ), the current article wishes to contribute insights into students' approaches to explore and solve problems related to minimal complex systems. By addressing research questions on the problem-solving strategies used, the study aims to understand students' exploration behavior in a complex problem-solving environment and the underlying causal relations. In this study, we show that such scoring can be developed through latent class analysis and that this alternative method of scoring may produce more reliable tests. Furthermore, such scoring can be automated and then employed in a large-scale assessment.

There are two major theoretical approaches to cognition relevant to our study; both offer general principles to interpret cognitive development beyond the narrower domain of problem solving. Piaget proposed the first comprehensive theory to explain the development of children's thinking as a sequence of four qualitatively different stages, the formal operational stage being the last one (Inhelder and Piaget, 1958 ), while the information processing approach describes human cognition by using terms and analogies borrowed from computer science. The information processing paradigm was not developed into an original developmental theory; it was rather aimed at reinterpreting and extending Piaget's theory (creating several Neo-Piagetian models) and synthesizing the main ideas of the two theoretical frameworks (Demetriou et al., 1993 ; Siegler, 1999 ). One of the focal points of these models is to explain the development of children's scientific reasoning, or, more closely, the way children understand how scientific experiments can be designed and how causal relationships can be explored by systematically changing the values of (independent) variables and observing their impact on other (target) variables.

From the perspective of the present study, the essential common element of cognitive developmental research is the control of variables strategy. Klahr and Dunbar ( 1988 ) distinguished two related skills in scientific thinking, hypothesis formation and experimental design, and they integrated these skills into a coherent model for a process of scientific discovery. The underlying assumption is that knowledge acquisition requires an iterative process involving both. System control as knowledge application tends to include both processes, especially when acquired knowledge turns out to be insufficient or dysfunctional (J. F. Beckmann, personal communication, August 16, 2017). Furthermore, they separated the processes of rule induction and problem solving, defining the latter as a search in a space of rules (Klahr and Dunbar, 1988 , p. 5).

de Jong and van Joolingen ( 1998 ) provided an overview of studies in scientific discovery learning with computer simulations. They concluded that a number of specific skills are needed for successful discovery, like systematic variation of variable values, which is in a focus of the present paper, and the use of high-quality heuristics for experimentation. They identified several characteristic problems in the discovery process and stressed that learners often have trouble interpreting data.

In one of the earliest systematic studies of students' problem-solving strategies, Vollmeyer et al. ( 1996 ) explored the impact of strategy systematicity and effectiveness on complex problem-solving performance. Based on previous studies, they distinguished the VOTAT strategy from other possible strategies [Change All (CA) and Heterogeneous (HT) other strategies], as VOTAT allows systematic exploration of the behavior of a system and a disconfirmation of hypotheses. In one of their experiments, they examined the hypothesis that VOTAT was more effective for acquiring knowledge than less systematic strategies. According to the results, the 36 undergraduate students had clearly shown strategy development. After interacting with the simulated system in several rounds, they tended to use the VOTAT strategy more frequently. In a second experiment, it was also demonstrated that goal specificity influences strategy use as well (Vollmeyer et al., 1996 ).

Beckmann and Goode ( 2014 ) analyzed the systematicity in exploration behavior in a study involving 80 first-year psychology students and focusing on the semantic context of a problem and its effect on the problem solvers' behavior in complex and dynamic systems. According to the results, a semantically familiar problem context invited a high number of a priori assumptions on the interdependency of system variables. These assumptions were less likely tested during the knowledge acquisition phase, this proving to be the main barrier to the acquisition of new knowledge. Unsystematic exploration behavior tended to produce non-informative system states that complicated the extraction of knowledge. A lack of knowledge ultimately led to poor control competency.

Beckmann et al. ( 2017 ) confirmed research results by Beckmann and Goode ( 2014 ) and demonstrated how a differentiation between complexity and difficulty leads to a better understanding of the cognitive mechanism behind CPS. According to findings from a study with 240 university students, the performance differences observed in the context of the semantic effect were associated with differences in the systematicity of the exploration behavior, and the systematicity of the exploration behavior was reflected in a specific sequence of interventions. They argued that it is only the VOTAT strategy—supplemented with the vary- none -at-a-time strategy in the case of noting autonomous changes—that creates informative system state transitions which enable problem solvers to derive knowledge of the causal structure of a complex, dynamic system.

Schoppek and Fischer ( 2017 ) also investigated VOTAT and the related “PULSE” strategy (all input variables to zero), which enables the problem solver to observe the eigendynamics of the system in a transfer experiment. They proposed that besides VOTAT and PULSE, other comprehensive knowledge elements and strategies, which contribute to successful CPS, should be investigated.

In a study with 2 nd - to 4 th -grade students, Chen and Klahr found little spontaneous development when children interacted with physical objects (in situations similar to that of Piaget's experiments), while more direct teaching of the control of variables strategy resulted in good effect sizes and older children were able to transfer the knowledge they had acquired (improved control of variable strategy) to remote contexts (Chen and Klahr, 1999 ). In a more recent study, Kuhn et al. ( 2008 ) further extended the scope of studies on scientific thinking, identifying three further aspects beyond the control of variables strategy, including coordinating effects of multiple influences, understanding the epistemological foundations of science and engaging in argumentation. In their experiment with 91 6th-grade students, they explored how students were able to estimate the impact of five independent variables simultaneously on a particular phenomenon, and they found that most students considered only one or two variables as possible causes.

In this paper, we explore several research questions on effective and less effective problem-solving strategies used in a complex problem-solving environment and detected by logfile analyses. We use logfile analyses to empirically test the success of different input behavior and strategy usage in CPS tasks within the MicroDYN framework. After constructing a mathematical model based on all theoretically effective strategies, which provide the problem solver with all the information needed to solve the problem, and defining several sub-strategies within the VOTAT strategy based on the amount of effort expended to extract the necessary information, we empirically distinguish different VOTAT and non-VOTAT strategies, which can result in good CPS performance and which go beyond the isolated variation strategy as an effective strategy for rule induction (Vollmeyer et al., 1996 ). We highlight the most and least effective VOTAT strategies used in solving MicroDYN problems and empirically investigate the awareness of the strategy used based on the sequence of the sub-strategies used. Based on these results, we conduct latent class analyses to explore and define patterns in qualitatively different VOTAT strategy uses.

We thus intend to answer five research questions:

• RQ1: Does the use of a theoretically effective strategy occur prior to high performance? In other words, does the use of a theoretically effective strategy result in high performance?
• RQ2: Do all VOTAT strategies result in a high CPS performance? What is the most effective VOTAT strategy?
• RQ3: How does awareness of the exploration strategy used influence overall performance on CPS tasks?
• RQ4: What profiles characterize the various problem solvers and explorers?
• RQ5: Do exploration strategy profiles differ across grade levels, which represent different educational stages during compulsory schooling?

In this study, we investigated qualitatively different classes of students' exploration behavior in CPS environments. We used latent class analysis (LCA) to study effective and non-effective input behavior and strategy use, especially the principle of isolated variation, across several CPS tasks. We compared the effectiveness of students' exploration behavior based on the amount of information they extracted with their problem-solving achievement. We posed five separate hypotheses.

Hypothesis 1: We expect that high problem-solving achievement is not closely related to expert exploration behavior.

Vollmeyer et al. ( 1996 ) explored the impact of strategy effectiveness on problem-solving performance and reported that effectiveness correlated negatively and weakly to moderately with solution error ( r = −0.32 and r = −0.54, p < 0.05). They reported that “most participants eventually adopted the most systematic strategy, VOTAT, and the more they used it, the better they tended to perform. However, even those using the VOTAT strategy generally did not solve the problem completely” (p. 88). Greiff et al. ( 2015 ) confirmed that different exploration behaviors are relevant to CPS and that the number of sub-strategies implemented was related to overall problem-solving achievement.

Hypothesis 2: We expect that students who use the isolated variation strategy in exploring CPS problems have a significantly better overall performance than those who use a theoretically effective, but different strategy.

Sonnleiter et al. ( 2017 ) noted that “A more effective exploration strategy leads to a higher system knowledge score and the higher the gathered knowledge, the better the ability to achieve the target values. Thus, system knowledge can be seen as a reliable and valid measure of students' mental problem representations” (p. 169). According to Wüstenberg et al. ( 2012 ), students who consistently apply the principle of isolated variation—the most systematic VOTAT strategy—in CPS environments show better overall CPS performance, compared to those who use different exploration strategies. Kröner et al. ( 2005 ) reported a positive correlation between using the principle of isolated variation and the likelihood of solving the overall problem.

Hypothesis 3: We expected that more aware CPS exploration behavior would be more effective than exploration behavior that generally results in extracting all the necessary information from the system to solve the problem, but within which the steps have no logically built structure and no systematicity in time.

Vollmeyer et al. ( 1996 ) explored the impact of strategy systematicity on problem-solving performance. They emphasized that “the systematicity of participants' spontaneous hypothesis-testing strategies predicted their success on learning the structure of the biology lab problem space” (p. 88). Vollmeyer and her colleagues restricted systematic strategy users to isolated variation strategy users; this corresponds to our terminology usage of aware isolated variation strategy users.

Hypothesis 4: We expected to find a distinct number of classes with statistically distinguishable profiles of CPS exploration behavior. Specifically, we expected to find classes of proficient, intermediate and low-performing explorers.

Several studies (Osman and Speekenbrink, 2011 ; Wüstenberg et al., 2012 ; Greiff et al., 2015 ) have indicated that there exist quantitative differences between different exploration strategies, which are relevant to a CPS environment. The current study is the first to investigate whether a relatively small number of qualitatively different profiles of students' exploration proficiency can be derived from their behavior detected in a CPS environment in a broad age range.

Hypothesis 5: We expected that more proficient CPS exploration behavior would be more dominant at later grade levels as an indication of cognitive maturation and of increasing abilities to explore CPS environments.

The cognitive development in children between Grades 3 and 12 is immense. According to Piaget's stage theory, they move from concrete operations to formal operations and they will be able to think logically and abstractly. According to Galotti ( 2011 ) and Molnár et al. ( 2013 ), the ability to solve problems effectively and to make decisions in CPS environments increases in this period of time; Grades 6–8 seem especially crucial for development. Thus, we expect that cognitive maturation will also be reflected in more proficient exploration behavior.

Participants

The sample was drawn from 3 rd - to 12 th -grade students (aged 9–18) in Hungarian primary and secondary schools ( N = 4,371; Table ​ Table1). 1 ). School classes formed the sampling unit. 180 classes from 50 schools in different regions were involved in the study, resulting in a wide-ranging distribution of students' background variables. The proportion of boys and girls was about the same.

Composition of samples.

The MicroDYN approach was employed to develop a measurement device for CPS. CPS tasks within the MicroDYN approach are based on linear structural equations (Funke, 2001 ), in which up to three input variables and up to three output variables are related (Greiff et al., 2013 ). Because of the small set of input and output variables, the MicroDYN problems could be understood completely with precise causal analyses (Funke, 2014 ). The relations are not presented to the problem solver in the scenario. To explore these relations, the problem solver must interact directly with the problem situation by manipulating the input variables (Greiff and Funke, 2010 ), an action that can influence the output variables (direct effects), and they must use the feedback provided by the computer to acquire and employ new knowledge (Fischer et al., 2012 ). Output variables can change spontaneously and can consist of internal dynamics, meaning they can change without changing the input variables (indirect effects; Greiff et al., 2013 ). Both direct and indirect effects can be detected with an adequate problem-solving strategy (Greiff et al., 2012 ). The interactions between the problem situation and the test taker play an important role, but they can only be identified in a computerized environment based on log data collected during test administration.

In this study, different versions with different levels of item complexity were used (Greiff et al., 2013 ), which varied by school grade (Table ​ (Table2; 2 ; six MicroDYN scenarios were administered in total in Grades 3–4; eight in Grade 5: nine in Grades 6–8; and twelve in Grades 9–12); however, we only involved those six tasks where the principle of isolated variation was the optimal exploration strategy. That is, we excluded problems with an external manipulation-independent, internal dynamic effect or multiple dependence effect from the analyses, and there were no delayed or accumulating effects used in the problem environments created. Complexity was defined by the number of input and output variables and the number of relations based on Cognitive Load Theory (Sweller, 1994 ). “Findings show that increases in the number of relations that must be processed in parallel in reasoning tasks consistently lead to increases in task difficulty” (Beckmann and Goode, 2017 ).

The design of the whole study: the complexity of the systems administered and the structure and anchoring of the tests applied in different grades.

The tasks were designed so that all causal relations could be identified with systematic manipulation of the inputs. The tasks contained up to three input variables and up to three output variables with different fictitious cover stories. The values of the input variables were changed by clicking on a button with a + or – sign or by using a slider connected to the respective variable (see Figure ​ Figure1). 1 ). The controllers of the input variables range from “– –” (value = −2) to “++” (value = +2). The history of the values of the input variables within the same scenario was presented on a graph connected to each input variable. Beyond the input and output variables, each scenario contained a Help, Reset, Apply and Next button. The Reset button set the system back to its original status. The Apply button made it possible to test the effect of the currently set values of the input variables on the output variables, which appeared in the form of a diagram of each output variable. According to the user interface, within the same phase of each of the problem scenarios, the input values remained at the level at which they were set for the previous input until the Reset button was pressed or they were changed manually. The Next button implemented the navigation between the different MicroDYN scenarios and the different phases within a MicroDYN scenario.

Exploration in phase 1 of the MicroDYN problems (two input variables and two output variables).

In the knowledge acquisition phase, participants were freely able to change the values of the input variables and attempt as many trials for each MicroDYN scenario as they liked within 180 s. During this 180 s, they had to draw the concept map (or causal diagram; Beckmann et al., 2017 ); that is, they had to draw the arrows between the variables presented on the concept map under the MicroDYN scenario on screen. In the knowledge application phase, students had to check their respective system using the right concept map presented on screen by reaching the given target values within a given time frame (90 s) in no more than four trials, that is, with a maximum of four clicks on the Apply button. This applied equally to all participants.

All of the CPS problems were administered online via the eDia platform. At the beginning, participants were provided with instructions about the usage of the user interface, including a warm-up task. Subsequently, participants had to explore, describe and operate unfamiliar systems. The assessment took place in the schools' ICT labs using the available school infrastructure. The whole CPS test took approximately 45 min to complete. Testing sessions were supervised by teachers who had been thoroughly trained in test administration. Students' problem-solving performance in the knowledge acquisition and application phases was automatically scored as CPS performance indicators; thus, problem solvers received immediate performance feedback at the end of the testing session. We split the sample into three age groups, whose achievement differed significantly (Grades 3–5, N = 1,871; Grades 6–7, N = 1,284; Grades 8–12, N = 1,216; F = 122.56, p < 0.001; t level_1_2 = −6.22, p < 0.001; t level_2_3 = −8.92, p < 0.001). This grouping corresponds to the changes in the developmental curve relevant to complex problem solving. The most intensive development takes place in Grades 6–7 (see Molnár et al., 2013 ). Measurement invariance, that is, the issue of structural stability, has been demonstrated with regard to complex problem solving in the MicroDYN approach already (e.g., Greiff et al., 2013 ) and was confirmed in the present study (Table ​ (Table3). 3 ). Between group differences can be interpreted as true and not as psychometric differences in latent ability. The comparisons across grade levels are valid.

Goodness of fit indices for measurement invariance of MicroDYN problems.

χ 2 and df were estimated by the weighted least squares mean and variance adjusted estimator (WLSMV). Δχ 2 and Δdf were estimated by the Difference Test procedure in MPlus. Chi-square differences between models cannot be compared by subtracting χ 2 s and dfs if WLSMV estimators are used. CFI, comparative fit index; TLI, Tucker Lewis index; RMSEA, root mean square error of approximation .

The latent class analysis (Collins and Lanza, 2010 ) employed in this study seeks students whose problem-solving strategies show similar patterns. It is a probabilistic or model-based technique, which is a variant of the traditional cluster analysis (Tein et al., 2013 ). The indicator variables observed were re-coded strategy scores. Robust maximum likelihood estimation was used and two to seven cluster solutions were examined. The process of latent class analysis is similar to that of cluster analysis. Information theory methods, likelihood ratio statistical test methods and entropy-based criteria were used in reducing the number of latent classes. As a measure of the relative model fit, AIC (Akaike Information Criterion), which considers the number of model parameters, and BIC (Bayesian Information Criterion), which considers the number of parameters and the number of observations, are the two original and most commonly used information theory methods for model selection. The adjusted Bayesian Information Criterion (aBIC) is the sample size-adjusted BIC. Lower values indicated a better model fit for each criterion (see Dziak et al., 2012 ). Entropy represents the precision of the classification for individual cases. MPlus reports the relative entropy index of the model, which is a re-scaled version of entropy on a [0,1] scale. Values near one, indicating high certainty in classification, and values near zero, indicating low certainty, both point to a low level of homogeneity of the clusters. Finally, the Lo–Mendell–Rubin Adjusted Likelihood Ratio Test (Lo et al., 2001 ) was employed to compare the model containing n latent classes with that containing n −1 latent classes. A significant p -value ( p < 0.05) indicates that the n −1 model is rejected in favor of a model with n classes, as it fits better than the previous one (Muthén and Muthén, 2012 ).

As previous research has found (Greiff et al., 2013 ), achievement in the first and second phases of the problem-solving process can be directly linked to the concept of knowledge acquisition (representation) and knowledge application (generating a solution) and was scored dichotomously. For knowledge acquisition, students' responses were scored as correct (“1”) if the connections between the variables were accurately indicated on the concept map (students' drawings fully matched the underlying problem structure); otherwise, the response was scored as incorrect (“0”). For knowledge application, students' responses were scored as correct (“1”) if students reached the given target values within a given time frame and in no more than four steps, that is, with a maximum of four clicks on the Apply button; otherwise, the response was scored as incorrect (“0”).

We developed a labeling procedure to divide the continuum of the problem-solving process into more scoreable phases and to score students' activity and behavior in the exploration phase at the beginning of the problem-solving process. For the different analyses and the most effective clustering, we applied a categorization, distinguishing students' use of the full, basic and minimal input behavior within a single CPS task (detailed description see later). The unit of this labeling process was a trial, a setting of the input variables, which was tested by clicking on the Apply button during the exploration phase of a problem, thus between receiving the problem and clicking on the Next button to reach the second part, the application part of the problem. The sum of these trials, within the same problem environment is called the input behavior. The input behavior was called a strategy if it followed meaningful regularities.

By our definition, the full input behavior model describes what exactly was done throughout the exploration phase and what kinds of trials were employed in the problem-solving process. It consists of all the activities with the sliders and Apply buttons in the order they were executed during the first phase, the exploration phase of the problem-solving process. The basic input behavior is part of the full input behavior model by definition, when the order of the trials attempted was still being taken into account, but it only consists of activities where students were able to acquire new information on the system. This means that the following activities and trials were not included in the basic input behavior model (they were deleted from the full input behavior model to obtain the basic behavior model):

• - where the same scenario, the same slider adjustment, was employed earlier within the task (that is, we excluded the role of ad hoc control behavior from the analyses),
• - where the value (position) of more than one input variable (slider) was changed and where the effect of the input variable on the operation of the system was still theoretically unknown to the problem solver,
• - where a new setting or new slider adjustment was employed, though the effect of the input variables used was known from previous settings.
• - As the basic input behavior involves timing, that is, the order of the trials used, it is suitable for the analyses with regard to the awareness of the input behavior employed.

Finally, we generated the students' minimal input behavior model from the full input behavior model. By our definition, the minimal input behavior focuses on those untimed activities (a simple list, without the real order of the trials), where students were able to obtain new information from the system and were able to do so by employing the most effective trials.

Each of the activities in which the students engaged and each of the trials which they used were labeled according to the following labeling system to be able to define students' full input behavior in a systematic format (please note that the numerical labels are neither scores nor ordinal or metric information):

• Only one single input variable was manipulated, whose relationship to the output variables was unknown (we considered a relationship unknown if its effect cannot be known from previous settings), while the other variables were set at a neutral value like zero. We labeled this trial +1.
• One single input variable was changed, whose relationship to the output variables was unknown. The others were not at zero, but at a setting used earlier. We labeled this trial +2.
• One single input variable was changed, whose relationship to the output variables was unknown, and the others were not at zero; however, the effect of the other input variable(s) was known from earlier settings. Even so, this combination was not attempted earlier. We labeled this trial +3.
• Everything was maintained in a neutral (zero) position. This trial is especially important for CPS problems with their own internal dynamics. We labeled this +A.
• The value of more than one input variable, whose relationship to the output variables was unknown, was changed at the same time, resulting in no additional information on the system. It was labeled –X.
• The same trial, the slider adjustment, had already been employed earlier within the task, resulting in no additional information on the system. It was labeled −0.
• A new slider adjustment was employed; however, the effect of the manipulated input variables was known from previous settings. This trial offered no additional information on the system and was labeled +0.

Although several input variables were changed by the scenario, it was theoretically possible to count the effect of the input variables on the output variables based on the information from the previous and present settings by using and solving linear equations. It was labeled +4.

An extra code (+5) was employed in the labeling process, but only for the basic input behavior, when the problem solver was able to figure out the structure of the problem based on the information obtained in the last trial used. This labeling has no meaning in the case of the minimal input behavior.

The full, basic and minimal input behavior models as well as the labeling procedure can be employed by analyzing problem solvers' exploration behavior and strategies for problems that are based on minimal complex systems. The user interface can preserve previous input values, and the values are not reset to zero after each exploration input. According to Fischer et al. ( 2012 ), VOTAT strategies are best for identifying causal relations between variables and they maximize the successful strategic behavior in minimal complex systems, such as CPS. By using a VOTAT strategy, the problem solver systematically varies only one input variable, while the others remain unchanged. This way, the effect of the changed variable can be found in the system by monitoring the changes in the output variables. There exist several types of VOTAT strategies based on the different combinations of VOTAT-centered trials +1, +2, and +3. The most obvious systematic strategy is when only one input variable is different from the neutral level in each trial and all the other input variables are systematically maintained at the neutral level. Thus, the strategy is a combination of so-called +1 trials, where it is employed for every input variable. Known as the isolated variation strategy (Müller et al., 2013 ), this strategy has been covered extensively in the literature. It must be noted that the isolated variation strategy is not appropriate to detect multiple dependence effects within the MicroDYN approach. We hypothesize that there are more and less successful input behaviors and strategies. We expect that theoretically effective, non-VOTAT strategies do not work as successfully as VOTAT strategies and that the most effective VOTAT strategy will be the isolated variation strategy.

We will illustrate the labeling and coding process and the course of generating a minimal input behavior out of a basic or full input behavior through the following two examples.

Figure ​ Figure1 1 shows an example with two input variables and two output variables. (The word problem reads as follows: “When you get home in the evening, there is a cat lying on your doorstep. It is exhausted and can barely move. You decide to feed it, and a neighbor gives you two kinds of cat food, Miaow and Catnip. Figure out how Miaow and Catnip impact activity and purring.”). The student who mapped the operation of the system as demonstrated in the figure pressed the Apply button six times in all, using the various settings for the Miaow and Catnip input variables.

In mapping the system, the problem solver kept the value of both the input variables at 0 in the first two steps (making no changes to the base values of the input variables), as a result of which the values of the output variables remained unchanged. In steps 3 and 4, he set the value of the Miaow input variable at 2, while the value of the Catnip variable remained at 0 (the bar chart by the name of each variable shows the history of these settings). Even making this change had no effect on the values of the output variables; that is, the values in each graph by the purring and activity variables are constantly horizontal. In steps 5 and 6, the student left the value of the Miaow input variable at 2, but a value of 2 was added to this for the Catnip input variable. As a result, the values of both output variables (purring and activity) began to grow by the same amount. The coding containing all the information (the full input behavior) for this sequence of steps was as follows: +A, −0, +1, −0, +2, −0. The reason for this is since steps 2, 4, and 6 were repetitions of previous combinations, we coded them as −0. Step 3 involved the purest use of a VOTAT strategy [changing the value of one input variable at a time, while keeping the values of the other input values at a neutral level (+1)], while the trial used in step 5 was also a VOTAT strategy. After all, only the value of one input variable changed compared to step 4. This is therefore not the same trial as we described in step 3 (+2). After step 5, all the necessary information was available to the problem solver. The basic input behavior for the same sequence of steps was +A, +1, +2, since the rest of the steps did not lead the problem solver to acquire unknown information. Independently of the time factor, the minimal input behavior in this case was also +A, +1, +2. The test taker was able to access new information on the operation of the system through these steps. From the point of view of awareness, this +1+2 strategy falls under aware strategy usage, as the +1 and +2 sub-strategies were not applied far apart (excluding the simple repetition of the executed trials next to each other) from each other in time. A good indicator of aware strategy usage is if there is no difference between minimal and basic input behavior.

In the second example (Figure ​ (Figure2), 2 ), we demonstrate the sequence of steps taken in mapping another problem as well as the coding we used. Here the students needed to solve a problem consisting of two input variables and one output variable. The word problem reads as follows: “Your mother has bought two new kinds of fruit drink mix. You want to make yourself a fruit drink with them. Figure out how the green and blue powders impact the sweetness of the drink. Plot your assumptions in the model.” The test taker attempted eight different trials in solving this problem, which were coded as follows: +1, +2, +0, +0, +0, +0, −0, −0. After step 2, the student had access to practically all the information required to plot the causal diagram. (In step 1, the problem solver checked the impact of one scoop of green powder and left the quantity of blue powder at zero. Once mixed, the resultant fruit drink became sweeter. In step 2, the problem solver likewise measured out one scoop of green powder for the drink but also added a scoop of blue powder. The sweetness of the drink changed as much as it had in step 1. After that, the student measured out various quantities of blue and then green powder, and looked at the impact.) The basic input behavior coded from the full input behavior used by the problem solver was +1+2, and the minimal input behavior was +1+1 because the purest VOTAT strategy was used in steps 1 and 6. (Thus, both variables separately confirmed the effects of the blue and the green powder on the sweetness of the drink.) From the point of view of awareness, this +1+1 strategy falls under non-aware strategy usage, as the two applications of the +1 trial occurred far apart from each other in time.

Exploration in phase 1 of the problems based on minimal complex systems (two input variables and one output variable).

Based on students' minimal input behavior we executed latent class analyses. We narrowed the focus to the principle of isolated variation, especially to the extent to which this special strategy was employed in the exploration phase as an indicator of students' ability to proficiently explore the problem environment. We added an extra variable to each of the problems, describing students' exploration behavior based on the following three categories: (1) no isolated variation at all (e.g., isolated variation was employed for none of the input variables – 0 points); (2) partially isolated variation (e.g., isolated variation was employed for some but not all the input variables – 1 point); and (3) fully isolated variation (e.g., isolated variation was employed for all the input variables – 2 points). Thus, depending on the level of optimal exploration strategy used, all the students received new categorical scores based on their input exploration behavior, one for each of the CPS tasks. Let us return to the example provided in Figures ​ Figures1, 1 , ​ ,2. 2 . In the first example, a partially isolated strategy was applied, since the problem solver only used this strategy to test the effect of the Miaow input variables (in trials 3 and 4). In the second example, a full isolated strategy was applied, as the problem solver used this isolated variation strategy for both the input variables during the exploration phase in the first and sixth trials.

The reliability of the test improved when scoring was based on the log data

The reliability of the MicroDYN problems as a measure of knowledge acquisition and knowledge application, the traditional CPS indicators for phases 1 and 2, were acceptable at α = 0.72–0.86 in all grades (Table ​ (Table4). 4 ). After we re-scored the problem solvers' behavior at the beginning of the problem-solving process, coded the log data and assigned new variables for the effectiveness of strategy usage during the exploration phase of the task for each task and person, the overall reliability of the test scores improved. This phenomenon was noted in all grades and in both coding procedures, when the amount of information obtained was examined (Cronbach's α ranged from 0.86 to 0.96) and when the level of optimal exploration strategy used was analyzed (Cronbach's α ranged from 0.83 to 0.98; the answers to the warm-up tasks were excluded from these analyses).

Internal consistencies in scoring the MicroDYN problems: analyses based on both traditional CPS indicators and re-coded log data based on student behavior at the beginning of the problem-solving process.

Use of a theoretically effective strategy does not result in high performance (RQ1)

Use of a theoretically effective strategy did not always result in high performance. The percentage of effective strategy use and high CPS performance varied from 20 to 80%, depending on the complexity of the CPS tasks and the age group. The percentage of theoretically effective strategy use in each cohort increased by 20% for age when problems with the same complexity were compared (Table ​ (Table5) 5 ) and decreased about 20% for the increasing number of input variables in the problems.

Percentage of theoretically effective and non-effective strategy use and high CPS performance.

The percentage of theoretically effective strategy use was the same for the less complex problems in Grades 3–5 and for the most complex tasks in Grades 8–12 (58%). More than 80% of these students solved the problem correctly in the first case, but only 60% had the correct solution in the second case. There was a 50% probability of effective and non-effective strategy use for problems with two input and two output variables in Grades 3–5 and for problems with three input and three output variables in Grades 6–7. In Grades 8–12, the use of a theoretically effective strategy was always higher than 50%, independently of the complexity of the problems (with no internal dynamic). The guessing factor, that is, the ad hoc optimization (use of a theoretically non-effective strategy with the correct solution) also changed, mostly based on the complexity and position of the tasks in the test. The results confirmed our hypothesis that the use of a theoretically effective strategy does not necessary represent the correct solution and that the correct solution does not always represent the use of an even theoretically effective problem-solving strategy.

Not all the VOTAT strategies result in high CPS performance (RQ2)

On average, only 15% of the theoretically effective strategy uses involved non-VOTAT strategies. The isolated variation strategy comprised 45% of the VOTAT strategies employed. It was the only theoretically effective strategy which always resulted in the correct solution to the problem with higher probability independently of problem complexity or the grade of the students. The real advantage of this strategy was most remarkable in the case of the third cohort, where an average of 80% of the students who employed this strategy solved the problems correctly (Figures ​ (Figures3, 3 , ​ ,4 4 ).

Efficacy of the most frequently employed VOTAT strategies on problems with two input variables and one or two output variables in Grades 3–5, 6–7, and 8–12.

Efficacy of the most frequently employed VOTAT strategies on problems with three input variables and one or two output variables in Grades 3–5, 6–7, and 8–12.

The second most frequently employed and successful VOTAT strategy was the +1+2 type or the +1+2+2 type, depending on the number of input variables. In the +1+2 type, only one single input variable was manipulated in the first step, while the other variable remained at a neutral value; in the second step, only the other input variable was changed and the first retained the setting used previously. This proved to be relatively successful on problems with a low level of complexity independently of age, but it generally resulted in a good solution with a low level of probability on more complex problems.

VOTAT strategies of the +1+3 type (in the case of two input variables) and of the +1+1+2 type (in the case of three input variables) were employed even less frequently and with a lower level of efficacy than all the other VOTAT strategies (+1+1+3, +1+2+1, +1+2+2, +1+2+3, +1+3+1, +1+3+2 and +1+3+3 in the case of three input variables) and theoretically effective, non-VOTAT strategies (e.g., +4 in the case of two input variables or +1+4, +4+2 and +4+3 in the case of three input variables). In the following, we provide an example of the +4+2 type, where the MicroDyn problem has three input variables (A, B, and C) and three output variables. In the first trial, the problem solver set the input variables to the following values: 0 (for variable A), 1 (for variable B), and 1 (for variable C); that is, he or she changed two input variables at the same time. In the second trial, he or she changed the value of two input variables at the same time again and applied the following setting: 0 (for variable A), −2 (for variable B), and −1 (for variable C). In the third trial, he set variable A to 1, and left variables B and C unchanged. That is, the problem solver's input behavior can be described with the following trials: –X +4 +2. Based on this strategy, it was possible to map the relationships between the input and output variables without using any VOTAT strategy in the exploration phase.

Aware explorers perform significantly higher on the CPS tasks (RQ3)

We compared the achievement of the aware, isolated strategy users with that of the non-aware explorers (Table ​ (Table6). 6 ). The percentage of high achievers among the non-aware explorers seemed to be almost independent of age, but strongly influenced by the complexity of the problem and the learning effect we noted in the testing procedure (see RQ5). Results for problems with two input variables and one output variable confirmed our previous results, which showed that the probability of providing the correct solution is very high even without aware use of a theoretically effective strategy (60–70%). With more complex problems, the difference between the percentages of aware and non-aware explorers was huge. Generally, 85% of the non-aware explorers failed on the problems, while at least 80% of the aware, isolated strategy users were able to solve the problems correctly.

Percentage of high achievers among aware and non-aware explorers by grade and problem complexity.

Six qualitatively different explorer class profiles can be distinguished at the end of the elementary level and five at the end of the secondary level (RQ4 and RQ5)

In all three cohorts, each of the information theory criteria used (AIC, BIC, and aBIC) indicated a continuous decrease in an increasing number of latent classes. The likelihood ratio statistical test (Lo–Mendell Rubin Adjusted Likelihood Ratio Test) showed the best model fit in Grades 3–5 for the 4-class model, in Grades 6–7 for the 6-class model and in Grades 8–12 for the 5-class model. The entropy-based criterion reached the maximum values for the 2- and 3-class solutions, but it was also high for the best-fitting models based on the information theory and likelihood ratio criteria. Thus, the entropy index for the 4-class model showed that 80% of the 3 rd - to 5 th -graders, 82% of the 6 th - to 7 th -graders and 85% of the 8 th - to 12 th -graders were accurately categorized based on their class membership (Table ​ (Table7 7 ).

Information theory, likelihood ratio and entropy-based fit indices for latent class analyses.

AIC, Akaike Information Criterion; BIC, Bayesian Information Criterion; aBIC, adjusted Bayesian Information Criterion; L–M–R test, Lo–Mendell–Rubin Adjusted Likelihood Ratio Test. The best fitting model solution is in italics .

We distinguished four latent classes in the lower grades based on the exploration strategy employed and the level of isolated variation strategy used (Table ​ (Table8): 8 ): 40.5% of the students proved to be non-performing explorers on the basis of their strategic patterns in the CPS environments. They did not use any isolated or partially isolated variation at all; 23.6% of the students were among the low-performing explorers who only rarely employed a fully or partially isolated variation strategy (with 0–20% probability on the less complex problems and 0–5% probability on the more complex problems). 24.7% of the 3 rd - to 5 th -graders were categorized as slow learners who were intermediate performers with regard to the efficiency of the exploration strategy they used on the easiest problems with a slow learning effect, but low-performing explorers on the complex ones. In addition, 11.1% of the students proved to be proficient explorers, who used the isolated or partially isolated variation strategy with 80–100% probability on all the proposed CPS problems.

Relative frequencies and average latent class probabilities across grade levels 3–5, 6–7, and 8–12.

In Grades 6–7, in which achievement proved to be significantly higher on average, 10% fewer students were observed in each of the first two classes (non-performing explorers and low-performing explorers). The percentage of intermediate explorers remained almost the same (26%), and we noted two more classes with the analyses: the class of rapid learners (4.4%) and that of slow learners, who are almost proficient explorers on the easiest problems, employing the fully or partially isolated variation strategy with 60–80% probability, but low-performing explorers on the complex ones (10.3%). The frequency of proficient strategy users was also increased (to 14.2%) compared to students in the lower grades. Finally, there was almost no change detected in the low-performing explorers' classes in Grades 8–12. We did not detect anyone in the class of intermediate explorers; they must have developed further and become (1) rapid learners (7.7%), (2) slow learners with almost high achievement with regard to the exploration strategy they used on the easiest problems, but low achievers on the complex ones (17.6%), or (3) proficient strategy users (26.3%), whose achievement was high both on the simplest and the most complex problems.

Based on these results, the percentage of non- and low explorers, who have very low exploration skills and do not learn during testing, decreased from almost 65 to 50% between the lower and higher primary school levels and then remained constant at the secondary level. There was a slight increase in respect of the percentage of students among the rapid learners. The students in that group used the fully or partially isolated strategy at very low levels at the beginning of the test, but they learned very quickly and detected these effective exploration strategies; thus, by the end of the test, their proficiency level with regard to exploration was equal to the top performers' achievement. However, we were unable to detect the class of rapid learners among 3 rd - to 5 th -graders.

Generally, students' level of exploration expertise with regard to fully and partially isolated variation improved significantly with age ( F = 70.376, p < 0.001). According to our expectations based on the achievement differences among students in Grades 3–5, 6–8 and 9–12, there were also significant differences in the level of expertise in fully or partially isolated strategy use during problem exploration between 3 rd - to 5 th - and 6 th - to 7 th -grade students ( t = −6.833, p < 0.001, d = 0.03) and between 6 th - to 7 th - and 8 th - to 12 th -grade students ( t = −6.993, p < 0.001, d = 0.03).

In this study, we examined 3 rd - to 12 th -grade (aged 9–18) students' problem-solving behavior by means of a logfile analysis to identify qualitatively different exploration strategies. Students' activity in the first phase of the problem-solving process was coded according to a mathematical model that was developed based on strategy effectiveness and then clustered for comparison. Reliability analyses of students' strategy use indicated that strategies used in the knowledge acquisition phase described students' development (ability level) better than traditional quantitative psychometric indicators, including the goodness of the model. The high reliability indices indicate that there are untapped possibilities in analyzing log data. Our analyses of logfiles extracted from a simulation-based assessment of problem solving have expanded the scope of previous studies and made it possible to identify a central component of children's scientific reasoning: the way students understand how scientific experiments can be designed and how causal relationships can be explored by systematically changing the values of (independent) variables and observing their impact on other (target) variables.

In this way, we have introduced a new labeling and scoring method that can be employed in addition to the two scores that have already been used in previous studies. We have found that using this scoring method (based on student strategy use) improves the reliability of the test. Further studies are needed to examine the validity of the scale based on this method and to determine what this scale really measures. We may assume that the general idea of varying the values of the independent variables and connecting them to the resultant changes in the target variable is the essence of scientific reasoning and that the systematic manipulation of variables is related to combinatorial reasoning, while summarizing one's observations and plotting a model is linked to rule induction. Such further studies have to place CPS testing in the context of other cognitive tests and may contribute to efforts to determine the place of CPS in a system of cognitive abilities (see e.g., Wüstenberg et al., 2012 ).

We have found that the use of a theoretically effective strategy does not always result in high performance. This is not surprising, and it confirms research results by de Jong and van Joolingen ( 1998 ), who argue that learners often have trouble interpreting data. As we observed earlier, using a systematic strategy requires combinatorial thinking, while drawing a conclusion from one's observations requires rule induction (inductive reasoning). Students showing systematic strategies but failing to solve the problem may possess combinatorial skills but lack the necessary level of inductive reasoning. It is more difficult to find an explanation for the other direction of discrepancy, when students actually solve the problem without an effective (complete) strategy. Thus, solving the problem does not require the use of a strategy which provides the problem solver with sufficient information about the problem environment to be able to form the correct solution. This finding is similar to results from previous research (e.g., Vollmeyer et al., 1996 ; Greiff et al., 2015 ). Goode and Beckmann ( 2010 ) reported two qualitatively different, but equally effective approaches: knowledge- based and ad hoc control.

In the present study, the contents of the problems were not based on real knowledge, and the causal relationships between the variables were artificial. Content knowledge was therefore no help to the students in filling the gap between the insufficient information acquired from interaction and the successful solution to the problem. We may assume that students guessed intuitively in such a case. Further studies may ascertain how students guess in such situations.

The percentage of success is influenced by the complexity of the CPS tasks, the type of theoretically effective strategy used, the age group and, finally, the degree to which the strategy was consciously employed.

The most frequently employed effective strategies fell within the class of VOTAT strategies. Almost half the VOTAT strategies were of the isolated variation strategy type, which resulted with higher probability in the correct solution independently of the complexity of the problem or the grade of the students. As noted earlier, not all the VOTAT strategies resulted in high CPS performance; moreover, all the other VOTAT strategies proved to be significantly less successful. Some of them worked with relative success on problems with a low level of complexity, but failed with a high level of probability on more complex problems independently of age group. Generally, the advantage of the isolated variation strategy (Wüstenberg et al., 2014 ) compared to the other VOTAT and non-VOTAT, theoretically effective strategies is clearly evident from the outcome. The use of the isolated variation strategy, where students examined the effect of the input variables on the output variables independently, resulted in a good solution with the highest probability and proved to be the most effective VOTAT strategy independently of student age or problem complexity.

Besides the type of strategy used, awareness also played an influential role. Aware VOTAT strategy users proved to be the most successful explorers. They were followed in effectiveness by non-aware VOTAT strategy users and theoretically effective, but non-VOTAT strategy users. They managed to represent the information that they had obtained from the system more effectively and made good decisions in the problem-solving process compared to their peers.

We noted both qualitative and quantitative changes of problem-solving behavior in the age range under examination. Using latent class analyses, we identified six qualitatively different class profiles during compulsory schooling. (1) Non-performing and (2) low-performing students who usually employed no fully or partially isolated variation strategy at all or, if so, then rarely. They basically demonstrated unsystematic exploration behavior. (3) Proficient strategy users who consistently employed optimal exploration strategies from the very first problem as well as the isolated variation strategy and the partially isolated variation, but only seldom. They must have more elaborated schemas available. (4) Slow learners who are intermediate performers on the easiest problems, but low performers on the complex ones or (5) high performers on the easiest problems, but low performers on the complex ones. Most members of this group managed to employ the principle of isolated or partially isolated variation and had an understanding of it, but they were only able to use it on the easiest task and then showed a rapid decline on the more complex CPS problems. They might have been cognitively overloaded by the increasingly difficult problem-solving environments they faced. (6) Rapid learners, a very small group from an educational point of view. These students started out as non-performers in their exploration behavior on the first CPS tasks, showed a rapid learning curve afterwards and began to use the partially isolated variation strategy increasingly and then the fully isolated variation strategy. By the end of the test, they reached the same high level of exploration behavior as the proficient explorers. We observed no so-called intermediate strategy users, i.e., those who used the partially isolated variation strategy almost exclusively on the test. As we expected, class membership increased significantly in the more proficient classes at the higher grade levels due to the effects of cognitive maturation and schooling, but this did not change noticeably in the two lowest-level classes.

Limitations of the study include the low sample size for secondary school students; further, repetition is required for validation. The generalizability of the results is also limited by the effects of semantic embedding (i.e., cover stories and variable labels), that is, the usage of different fictitious cover stories “with the intention of minimizing the uncontrollable effects of prior knowledge, beliefs or suppositions” (Beckmann and Goode, 2017 ). An assumption triggered by semantic contexts has an impact on exploration behavior (e.g., the range of interventions, or strategies employed by the problem solver; Beckmann and Goode, 2014 ), that is, how the problem solver interacts with the system. Limitations also include the characteristics of the interface used. In our view, analyses with regard to VOTAT strategies are only meaningful in systems with an interface where inputs do not automatically reset to zero from one input to the next (Beckmann and Goode, 2017 ). That is, we excluded problem environments from the study where the inputs automatically reset to zero from one input to the next. A further limitation of the generalizability of the results is that we have omitted problems with autonomic changes from the analyses.

The main reason why we have excluded systems that contain autoregressive dependencies from the analyses is that different strategy usage is required on problems which also involve the use of trial +A (according to our coding of sub-strategies), which is not among the effective sub-strategies for problems without autonomic changes. Analyses of students' behavior on problems with autonomic changes will form part of further studies, as well as a refinement of the definition of what makes a problem complex and difficult. We plan to adapt the Person, Task and Situation framework published by Beckmann and Goode ( 2017 ). The role of ad hoc control behavior was excluded from the analyses; further studies are required to ascertain the importance of the repetitive control behavior. Another limitation of the study could be the interpretation of the differences across age group clusters as indicators of development and not as a lack of stability of the model employed.

These results shed new light on and provide a new interpretation of previous analyses of complex problem solving in the MicroDYN approach. They also highlight the importance of explicit enhancement of problem-solving skills and problem-solving strategies as a tool for applying knowledge in a new context during school lessons.

Ethics statement

Ethical approval was not required for this study based on national and institutional guidelines. The assessments which provided data for this study were integrated parts of the educational processes of the participating schools. The coding system for the online platform masked students' identity; the data cannot be connected to the students. The results from the no-stakes diagnostic assessments were disclosed only to the participating students (as immediate feedback) and to their teachers. Because of the anonymity and no-stakes testing design of the assessment process, it was not required or possible to request and obtain written informed parental consent from the participants.

Author contributions

Both the authors, GM and BC, certify that they have participated sufficiently in the work to take responsibility for the content, including participation in the concept, design and analysis as well as the writing and final approval of the manuscript. Each author agrees to be accountable for all aspects of the work.

Conflict of interest statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

1 With regard to terminology, please note that different terms are used for the subject at hand (e.g., complex problem solving, dynamic problem solving, interactive problem solving and creative problem solving). In this paper, we use the modifier “complex” (see Csapó and Funke, 2017 ; Dörner and Funke, 2017 ).

Funding. This study was funded by OTKA K115497.

• OECD (2014). PISA 2012 Results: Creative Problem Solving. Students' Skills in Tackling Real-Life Problems (Volume V) . Paris: OECD. [ Google Scholar ]
• Beckmann J. F., Goode N. (2014). The benefit of being näive and knowing it? The unfavourable impact of perceived context familiarity on learning in complex problem solving tasks . Instructional Sci . 42 , 271–290. 10.1007/s11251-013-9280-7 [ CrossRef ] [ Google Scholar ]
• Beckmann J. F., Birney D. P., Goode N. (2017). Beyond psychometrics: the difference between difficult problem solving and complex problem solving . Front. Psychol . 8 :1739. 10.3389/fpsyg.2017.01739 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Beckmann J. F., Goode N. (2017). Missing the wood for the wrong trees: On the difficulty of defining the complexity of complex problem solving scenarios . J. Intell . 5 :2 10.3390/jintelligence5020015 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chen Z., Klahr D. (1999). All other things being equal: acquisition and transfer of the control of variables strategy . Child Dev. 70 , 1098–1120. 10.1111/1467-8624.00081 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Collins L. M., Lanza S. T. (2010). Latent Class and Latent Transition Analysis: With Applications in the Social, Behavioral, and Health Sciences . New York, NY: Wiley. [ Google Scholar ]
• Csapó B., Ainley J., Bennett R., Latour T., Law N. (2012). Technological issues of computer-based assessment of 21st century skills , in Assessment and Teaching of 21st Century Skills , eds Griffin P., McGaw B., Care E. (New York, NY: Springer; ), 143–230. [ Google Scholar ]
• Csapó B., Funke J. (eds.). (2017). The Nature of Problem Solving. Using Research to Inspire 21st Century Learning. Paris: OECD. [ Google Scholar ]
• Csapó B., Molnár G. (2017). Potential for assessing dynamic problem-solving at the beginning of higher education studies . Front. Psychol. 8 :2022. 10.3389/fpsyg.2017.02022 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Csapó B., Molnár G., Nagy J. (2014). Computer-based assessment of school-readiness and reasoning skills . J. Educ. Psychol. 106 , 639–650. 10.1037/a0035756 [ CrossRef ] [ Google Scholar ]
• de Jong T., van Joolingen W. R. (1998). Scientific discovery learning with computer simulations of conceptual domains . Rev. Educ. Res . 68 , 179–201. 10.3102/00346543068002179 [ CrossRef ] [ Google Scholar ]
• Demetriou A., Efklides A., Platsidou M. (1993). The architecture and dynamics of developing mind: experiential structuralism as a frame for unifying cognitive developmental theories . Monogr. Soc. Res. Child Dev. 58 , 1–205. 10.2307/1166053 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Dörner D., Funke J. (2017). Complex problem solving: what it is and what it is not . Front. Psychol. 8 :1153 10.3389/fpsyg.2017.01153 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Dziak J. J., Coffman D. L., Lanza S. T., Li R. (2012). Sensitivity and Specificity of Information Criteria . The Pennsylvania State University: The Methodology Center and Department of Statistics. Available online at: https://methodology.psu.edu/media/techreports/12-119.pdf
• Fischer A., Greiff S., Funke J. (2012). The process of solving complex problems . J. Probl. Solving . 4 , 19–42. 10.7771/1932-6246.1118 [ CrossRef ] [ Google Scholar ]
• Funke J. (2001). Dynamic systems as tools for analysing human judgement . Think. Reason. 7 , 69–89. 10.1080/13546780042000046 [ CrossRef ] [ Google Scholar ]
• Funke J. (2010). Complex problem solving: a case for complex cognition? Cogn. Process. 11 , 133–142. 10.1007/s10339-009-0345-0 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Funke J. (2014). Analysis of minimal complex systems and complex problem solving require different forms of causal cognition . Front. Psychol. 5 :739. 10.3389/fpsyg.2014.00739 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Galotti K. M. (2011). Cognitive Development . Thousand Oaks, CA: SAGE. [ Google Scholar ]
• Gardner P. H., Berry D. C. (1995). The effect of different forms of advice on the control of a simulated complex system . Appl. Cogn. Psychol. 9 :7 10.1002/acp.2350090706 [ CrossRef ] [ Google Scholar ]
• Goode N., Beckmann J. F. (2010). You need to know: there is a causal relationship between structural knowledge and control performance in complex problem solving tasks . Intelligence 38 , 345–352. 10.1016/j.intell.2010.01.001 [ CrossRef ] [ Google Scholar ]
• Greiff S., Funke J. (2010). Systematische Erforschung komplexer Problemlösefähigkeit anhand minimal komplexer Systeme . Zeitschrift für Pädagogik 56 , 216–227. [ Google Scholar ]
• Greiff S., Funke J. (2017). Interactive problem solving: exploring the potential of minimal complex systems , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (Paris: OECD; ), 93–105. [ Google Scholar ]
• Greiff S., Wüstenberg S., Avvisati F. (2015). Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving . Computers Educ . 91 , 92–105. 10.1016/j.compedu.2015.10.018 [ CrossRef ] [ Google Scholar ]
• Greiff S., Wüstenberg S., Csapó B., Demetriou A., Hautamäki H., Graesser A. C., et al. (2014). Domain-general problem solving skills and education in the 21st century . Educ. Res. Rev . 13 , 74–83. 10.1016/j.edurev.2014.10.002 [ CrossRef ] [ Google Scholar ]
• Greiff S., Wüstenberg S., Funke J. (2012). Dynamic problem solving: a new assessment perspective . Appl. Psychol. Meas. 36 , 189–213. 10.1177/0146621612439620 [ CrossRef ] [ Google Scholar ]
• Greiff S., Wüstenberg S., Goetz T., Vainikainen M.-P., Hautamäki J., Bornstein M. H. (2015). A longitudinal study of higher-order thinking skills: working memory and fluid reasoning in childhood enhance complex problem solving in adolescence . Front. Psychol. 6 :1060. 10.3389/fpsyg.2015.01060 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Greiff S., Wüstenberg S., Molnár G., Fischer A., Funke J., Csapó B. (2013). Complex problem solving in educational contexts – Something beyond g: Concept, assessment, measurement invariance, and construct validity . J. Educ. Psychol . 105 , 364–379. 10.1037/a0031856 [ CrossRef ] [ Google Scholar ]
• Griffin P., McGaw B., Care E. (2012). Assessment and Teaching of 21st Century Skills . Dordrecht: Springer. [ Google Scholar ]
• Inhelder B., Piaget J. (1958). The Growth of Logical Thinking from Childhood to Adolescence . New York, NY: Basic Books. [ Google Scholar ]
• Klahr D., Dunbar K. (1988). Dual space search during scientific reasoning . Cogn. Sci. 12 , 1–48. 10.1207/s15516709cog1201_1 [ CrossRef ] [ Google Scholar ]
• Klahr D., Triona L. M., Williams C. (2007). Hands on what? The relative effectiveness of physical versus virtual materials in an engineering design project by middle school children . J. Res. Sci. Teaching 44 , 183–203. 10.1002/tea.20152 [ CrossRef ] [ Google Scholar ]
• Kröner S., Plass J. L., Leutner D. (2005). Intelligence assessment with computer simulations . Intelligence 33 , 347–368. 10.1016/j.intell.2005.03.002 [ CrossRef ] [ Google Scholar ]
• Kuhn D., Iordanou K., Pease M., Wirkala C. (2008). Beyond control of variables: what needs to develop to achieve skilled scientific thinking? Cogn. Dev . 23 , 435–451. 10.1016/j.cogdev.2008.09.006 [ CrossRef ] [ Google Scholar ]
• Lo Y., Mendell N. R., Rubin D. B. (2001). Testing the number of components in a normal mixture . Biometrika 88 , 767–778. 10.1093/biomet/88.3.767 [ CrossRef ] [ Google Scholar ]
• Lotz C., Scherer R., Greiff S., Sparfeldt J. R. (2017). Intelligence in action – Effective strategic behaviors while solving complex problems . Intelligence 64 , 98–112. 10.1016/j.intell.2017.08.002 [ CrossRef ] [ Google Scholar ]
• Mayer R. E., Wittrock M. C. (1996). Problem-solving transfer in Handbook of Educational Psychology , eds Berliner D. C., Calfee R. C. (New York, NY; London: Routledge; ), 47–62. [ Google Scholar ]
• Molnár G., Greiff S., Csapó B. (2013). Inductive reasoning, domain specific and complex problem solving: relations and development . Think. Skills Creat. 9 , 35–45. 10.1016/j.tsc.2013.03.002 [ CrossRef ] [ Google Scholar ]
• Molnár G., Greiff S., Wüstenberg S., Fischer A. (2017). Empirical study of computer-based assessment of domain-general complex problem-solving skills , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (Paris: OECD; ), 125–140. [ Google Scholar ]
• Müller J. C., Kretzschmar A., Greiff S. (2013). Exploring exploration: Inquiries into exploration behavior in complex problem solving assessment , in Proceedings of the 6th International Conference on Educational Data Mining , eds D'Mello S. K., Calvo R. A., Olney A. 336–337. Available online at: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.666.6664&rep=rep1&type=pdf [ Google Scholar ]
• Muthén L. K., Muthén B. O. (2012). Mplus User's Guide, 7th Edn . Los Angeles, CA: Muthén and Muthén. [ Google Scholar ]
• Osman M., Speekenbrink M. (2011). Cue utilization and strategy application in stable and unstable dynamic environments . Cogn. Syst. Res . 12 , 355–364. 10.1016/j.cogsys.2010.12.004 [ CrossRef ] [ Google Scholar ]
• Schoppek W., Fischer A. (2017). Common process demands of two complex dynamic control tasks: transfer is mediated by comprehensive strategies . Front. Psychol . 8 :2145. 10.3389/fpsyg.2017.02145 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Siegler R. S. (1999). Strategic development . Trends Cogn. Sci. 3 , 430–435. [ PubMed ] [ Google Scholar ]
• Sonnleiter P., Keller U., Martin R., Latour T., Brunner M. (2017). Assessing complex problem solving in the classroom: meeting challenges and opportunities , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (Paris: OECD; ), 159–174. [ Google Scholar ]
• Sweller J. (1994). Cognitive load theory, learning difficulty, and instructional design . Learn. Instruct . 4 , 295–312. 10.1016/0959-4752(94)90003-5 [ CrossRef ] [ Google Scholar ]
• Tein J. Y., Coxe S., Cham H. (2013). Statistical power to detect the correct number of classes in latent profile analysis . Struct. Equ. Modeling. 20 , 640–657. 10.1080/10705511.2013.824781 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Tóth K., Rölke H., Goldhammer F., Barkow I. (2017). Educational process mining: New possibilities for understanding students' problem-solving skills , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds. Csapó B., Funke J. (Paris: OECD; ), 193–210. [ Google Scholar ]
• Tschirgi J. E. (1980). Sensible reasoning: a hypothesis about hypotheses . Child Dev. 51 , 1–10. 10.2307/1129583 [ CrossRef ] [ Google Scholar ]
• Vollmeyer R., Burns B. D., Holyoak K. J. (1996). The impact of goal specificity on strategy use and the acquisition of problem structure . Cogn. Sci. 20 , 75–100. 10.1207/s15516709cog2001_3 [ CrossRef ] [ Google Scholar ]
• Wüstenberg S., Greiff S., Funke J. (2012). Complex problem solving. More than reasoning? Intelligence 40 , 1–14. 10.1016/j.intell.2011.11.003 [ CrossRef ] [ Google Scholar ]
• Wüstenberg S., Stadler M., Hautamäki J., Greiff S. (2014). The role of strategy knowledge for the application of strategies in complex problem solving tasks . Technol. Knowledge Learn. 19 , 127–146. 10.1007/s10758-014-9222-8 [ CrossRef ] [ Google Scholar ]
• Zoanetti N., Griffin P. (2017). Log-file data as indicators for problem-solving processes , in The Nature of Problem Solving. Using Research to Inspire 21st Century Learning , eds Csapó B., Funke J. (Paris: OECD; ), 177–191. [ Google Scholar ]

STEM Problem Solving: Inquiry, Concepts, and Reasoning

• Published: 29 January 2022
• Volume 32 , pages 381–397, ( 2023 )

Cite this article

• Aik-Ling Tan   ORCID: orcid.org/0000-0002-4627-4977 1 ,
• Yann Shiou Ong   ORCID: orcid.org/0000-0002-6092-2803 1 ,
• Yong Sim Ng   ORCID: orcid.org/0000-0002-8400-2040 1 &
• Jared Hong Jie Tan 1  

10k Accesses

8 Citations

2 Altmetric

Explore all metrics

Balancing disciplinary knowledge and practical reasoning in problem solving is needed for meaningful learning. In STEM problem solving, science subject matter with associated practices often appears distant to learners due to its abstract nature. Consequently, learners experience difficulties making meaningful connections between science and their daily experiences. Applying Dewey’s idea of practical and science inquiry and Bereiter’s idea of referent-centred and problem-centred knowledge, we examine how integrated STEM problem solving offers opportunities for learners to shuttle between practical and science inquiry and the kinds of knowledge that result from each form of inquiry. We hypothesize that connecting science inquiry with practical inquiry narrows the gap between science and everyday experiences to overcome isolation and fragmentation of science learning. In this study, we examine classroom talk as students engage in problem solving to increase crop yield. Qualitative content analysis of the utterances of six classes of 113 eighth graders and their teachers were conducted for 3 hours of video recordings. Analysis showed an almost equal amount of science and practical inquiry talk. Teachers and students applied their everyday experiences to generate solutions. Science talk was at the basic level of facts and was used to explain reasons for specific design considerations. There was little evidence of higher-level scientific conceptual knowledge being applied. Our observations suggest opportunities for more intentional connections of science to practical problem solving, if we intend to apply higher-order scientific knowledge in problem solving. Deliberate application and reference to scientific knowledge could improve the quality of solutions generated.

Similar content being viewed by others

Science Camps for Introducing Nature of Scientific Inquiry Through Student Inquiries in Nature: Two Applications with Retention Study

Ways of thinking in STEM-based problem solving

Framing and Assessing Scientific Inquiry Practices

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

As we enter to second quarter of the twenty-first century, it is timely to take stock of both the changes and demands that continue to weigh on our education system. A recent report by World Economic Forum highlighted the need to continuously re-position and re-invent education to meet the challenges presented by the disruptions brought upon by the fourth industrial revolution (World Economic Forum, 2020 ). There is increasing pressure for education to equip children with the necessary, relevant, and meaningful knowledge, skills, and attitudes to create a “more inclusive, cohesive and productive world” (World Economic Forum, 2020 , p. 4). Further, the shift in emphasis towards twenty-first century competencies over mere acquisition of disciplinary content knowledge is more urgent since we are preparing students for “jobs that do not yet exist, technology that has not yet been invented, and problems that has yet exist” (OECD, 2018 , p. 2). Tan ( 2020 ) concurred with the urgent need to extend the focus of education, particularly in science education, such that learners can learn to think differently about possibilities in this world. Amidst this rhetoric for change, the questions that remained to be answered include how can science education transform itself to be more relevant; what is the role that science education play in integrated STEM learning; how can scientific knowledge, skills and epistemic practices of science be infused in integrated STEM learning; what kinds of STEM problems should we expose students to for them to learn disciplinary knowledge and skills; and what is the relationship between learning disciplinary content knowledge and problem solving skills?

In seeking to understand the extent of science learning that took place within integrated STEM learning, we dissected the STEM problems that were presented to students and examined in detail the sense making processes that students utilized when they worked on the problems. We adopted Dewey’s ( 1938 ) theoretical idea of scientific and practical/common-sense inquiry and Bereiter’s ideas of referent-centred and problem-centred knowledge building process to interpret teacher-students’ interactions during problem solving. There are two primary reasons for choosing these two theoretical frameworks. Firstly, Dewey’s ideas about the relationship between science inquiry and every day practical problem-solving is important in helping us understand the role of science subject matter knowledge and science inquiry in solving practical real-world problems that are commonly used in STEM learning. Secondly, Bereiter’s ideas of referent-centred and problem-centred knowledge augment our understanding of the types of knowledge that students can learn when they engage in solving practical real-world problems.

Taken together, Dewey’s and Bereiter’s ideas enable us to better understand the types of problems used in STEM learning and their corresponding knowledge that is privileged during the problem-solving process. As such, the two theoretical lenses offered an alternative and convincing way to understand the actual types of knowledge that are used within the context of integrated STEM and help to move our understanding of STEM learning beyond current focus on examining how engineering can be used as an integrative mechanism (Bryan et al., 2016 ) or applying the argument of the strengths of trans-, multi-, or inter-disciplinary activities (Bybee, 2013 ; Park et al., 2020 ) or mapping problems by the content and context as pure STEM problems, STEM-related problems or non-STEM problems (Pleasants, 2020 ). Further, existing research (for example, Gale et al., 2000 ) around STEM education focussed largely on description of students’ learning experiences with insufficient attention given to the connections between disciplinary conceptual knowledge and inquiry processes that students use to arrive at solutions to problems. Clarity in the role of disciplinary knowledge and the related inquiry will allow for more intentional design of STEM problems for students to learn higher-order knowledge. Applying Dewey’s idea of practical and scientific inquiry and Bereiter’s ideas of referent-centred and problem-centred knowledge, we analysed six lessons where students engaged with integrated STEM problem solving to propose answers to the following research questions: What is the extent of practical and scientific inquiry in integrated STEM problem solving? and What conceptual knowledge and problem-solving skills are learnt through practical and science inquiry during integrated STEM problem solving?

2 Inquiry in Problem Solving

Inquiry, according to Dewey ( 1938 ), involves the direct control of unknown situations to change them into a coherent and unified one. Inquiry usually encompasses two interrelated activities—(1) thinking about ideas related to conceptual subject-matter and (2) engaging in activities involving our senses or using specific observational techniques. The National Science Education Standards released by the National Research Council in the US in 1996 defined inquiry as “…a multifaceted activity that involves making observations; posing questions; examining books and other sources of information to see what is already known; planning investigations; reviewing what is already known in light of experimental evidence; using tools to gather, analyze, and interpret data; proposing answers, explanations, and predictions; and communicating the results. Inquiry requires identification of assumptions, use of critical and logical thinking, and consideration of alternative explanations” (p. 23). Planning investigation; collecting empirical evidence; using tools to gather, analyse and interpret data; and reasoning are common processes shared in the field of science and engineering and hence are highly relevant to apply to integrated STEM education.

In STEM education, establishing the connection between general inquiry and its application helps to link disciplinary understanding to epistemic knowledge. For instance, methods of science inquiry are popular in STEM education due to the familiarity that teachers have with scientific methods. Science inquiry, a specific form of inquiry, has appeared in many science curriculum (e.g. NRC, 2000 ) since Dewey proposed in 1910 that learning of science should be perceived as both subject-matter and a method of learning science (Dewey, 1910a , 1910b ). Science inquiry which involved ways of doing science should also encompass the ways in which students learn the scientific knowledge and investigative methods that enable scientific knowledge to be constructed. Asking scientifically orientated questions, collecting empirical evidence, crafting explanations, proposing models and reasoning based on available evidence are affordances of scientific inquiry. As such, science should be pursued as a way of knowing rather than merely acquisition of scientific knowledge.

Building on these affordances of science inquiry, Duschl and Bybee ( 2014 ) advocated the 5D model that focused on the practice of planning and carrying out investigations in science and engineering, representing two of the four disciplines in STEM. The 5D model includes science inquiry aspects such as (1) deciding on what and how to measure, observe and sample; (2) developing and selecting appropriate tools to measure and collect data; (3) recording the results and observations in a systematic manner; (4) creating ways to represent the data and patterns that are observed; and (5) determining the validity and the representativeness of the data collected. The focus on planning and carrying out investigations in the 5D model is used to help teachers bridge the gap between the practices of building and refining models and explanation in science and engineering. Indeed, a common approach to incorporating science inquiry in integrated STEM curriculum involves student planning and carrying out scientific investigations and making sense of the data collected to inform engineering design solution (Cunningham & Lachapelle, 2016 ; Roehrig et al., 2021 ). Duschl and Bybee ( 2014 ) argued that it is needful to design experiences for learners to appreciate that struggles are part of problem solving in science and engineering. They argued that “when the struggles of doing science is eliminated or simplified, learners get the wrong perceptions of what is involved when obtaining scientific knowledge and evidence” (Duschl & Bybee, 2014 , p. 2). While we concur with Duschl and Bybee about the need for struggles, in STEM learning, these struggles must be purposeful and grade appropriate so that students will also be able to experience success amidst failure.

The peculiar nature of science inquiry was scrutinized by Dewey ( 1938 ) when he cross-examined the relationship between science inquiry and other forms of inquiry, particularly common-sense inquiry. He positioned science inquiry along a continuum with general or common-sense inquiry that he termed as “logic”. Dewey argued that common-sense inquiry serves a practical purpose and exhibits features of science inquiry such as asking questions and a reliance on evidence although the focus of common-sense inquiry tends to be different. Common-sense inquiry deals with issues or problems that are in the immediate environment where people live, whereas the objects of science inquiry are more likely to be distant (e.g. spintronics) from familiar experiences in people’s daily lives. While we acknowledge the fundamental differences (such as novel discovery compared with re-discovering science, ‘messy’ science compared with ‘sanitised’ science) between school science and science that is practiced by scientists, the subject of interest in science (understanding the world around us) remains the same.

The unfamiliarity between the functionality and purpose of science inquiry to improve the daily lives of learners does little to motivate learners to learn science (Aikenhead, 2006 ; Lee & Luykx, 2006 ) since learners may not appreciate the connections of science inquiry in their day-to-day needs and wants. Bereiter ( 1992 ) has also distinguished knowledge into two forms—referent-centred and problem-centred. Referent-centred knowledge refers to subject-matter that is organised around topics such as that in textbooks. Problem-centred knowledge is knowledge that is organised around problems, whether they are transient problems, practical problems or problems of explanations. Bereiter argued that referent-centred knowledge that is commonly taught in schools is limited in their applications and meaningfulness to the lives of students. This lack of familiarity and affinity to referent-centred knowledge is likened to the science subject-matter knowledge that was mentioned by Dewey. Rather, it is problem-centred knowledge that would be useful when students encounter problems. Learning problem-centred knowledge will allow learners to readily harness the relevant knowledge base that is useful to understand and solve specific problems. This suggests a need to help learners make the meaningful connections between science and their daily lives.

Further, Dewey opined that while the contexts in which scientific knowledge arise could be different from our daily common-sense world, careful consideration of scientific activities and applying the resultant knowledge to daily situations for use and enjoyment is possible. Similarly, in arguing for problem-centred knowledge, Bereiter ( 1992 ) questioned the value of inert knowledge that plays no role in helping us understand or deal with the world around us. Referent-centred knowledge has a higher tendency to be inert due to the way that the knowledge is organised and the way that the knowledge is encountered by learners. For instance, learning about the equation and conditions for photosynthesis is not going to help learners appreciate how plants are adapted for photosynthesis and how these adaptations can allow plants to survive changes in climate and for farmers to grow plants better by creating the best growing conditions. Rather, students could be exposed to problems of explanations where they are asked to unravel the possible reasons for low crop yield and suggest possible ways to overcome the problem. Hence, we argue here that the value of the referent knowledge is that they form the basis and foundation for the students to be able to discuss or suggest ways to overcome real life problems. Referent-centred knowledge serves as part of the relevant knowledge base that can be harnessed to solve specific problems or as foundational knowledge students need to progress to learn higher-order conceptual knowledge that typically forms the foundations or pillars within a discipline. This notion of referent-centred knowledge serving as foundational knowledge that can be and should be activated for application in problem-solving situation is shown by Delahunty et al. ( 2020 ). They found that students show high reliance on memory when they are conceptualising convergent problem-solving tasks.

While Bereiter argues for problem-centred knowledge, he cautioned that engagement should be with problems of explanation rather than transient or practical problems. He opined that if learners only engage in transient or practical problem alone, they will only learn basic-category types of knowledge and fail to understand higher-order conceptual knowledge. For example, for photosynthesis, basic-level types of knowledge included facts about the conditions required for photosynthesis, listing the products formed from the process of photosynthesis and knowing that green leaves reflect green light. These basic-level knowledges should intentionally help learners learn higher-level conceptual knowledge that include learners being able to draw on the conditions for photosynthesis when they encounter that a plant is not growing well or is exhibiting discoloration of leaves.

Transient problems disappear once a solution becomes available and there is a high likelihood that we will not remember the problem after that. Practical problems, according to Bereiter are “stuck-door” problems that could be solved with or without basic-level knowledge and often have solutions that lacks precise definition. There are usually a handful of practical strategies, such as pulling or pushing the door harder, kicking the door, etc. that will work for the problems. All these solutions lack a well-defined approach related to general scientific principles that are reproducible. Problems of explanations are the most desirable types of problems for learners since these are problems that persist and recur such that they can become organising points for knowledge. Problems of explanations consist of the conceptual representations of (1) a text base that serves to represent the text content and (2) a situation model that shows the portion of the world in which the text is relevant. The idea of text base to represent text content in solving problems of explanations is like the idea of domain knowledge and structural knowledge (refers to knowledge of how concepts within a domain are connected) proposed by Jonassen ( 2000 ). He argued that both types of knowledges are required to solve a range of problems from well-structured problems to ill-structured problems with a simulated context, to simple ill-structured problems and to complex ill-structured problems.

Jonassen indicated that complex ill-structured problems are typically design problems and are likely to be the most useful forms of problems for learners to be engaged in inquiry. Complex ill-structured design problems are the “wicked” problems that Buchanan ( 1992 ) discussed. Buchanan’s idea is that design aims to incorporate knowledge from different fields of specialised inquiry to become whole. Complex or wicked problems are akin to the work of scientists who navigate multiple factors and evidence to offer models that are typically oversimplified, but they apply them to propose possible first approximation explanations or solutions and iteratively relax constraints or assumptions to refine the model. The connections between the subject matter of science and the design process to engineer a solution are delicate. While it is important to ensure that practical concerns and questions are taken into consideration in designing solutions (particularly a material artefact) to a practical problem, the challenge here lies in ensuring that creativity in design is encouraged even if students initially lack or neglect the scientific conceptual understanding to explain/justify their design. In his articulation of wicked problems and the role of design thinking, Buchanan ( 1992 ) highlighted the need to pay attention to category and placement. Categories “have fixed meanings that are accepted within the framework of a theory or a philosophy and serve as the basis for analyzing what already exist” (Buchanan, 1992 , p. 12). Placements, on the other hand, “have boundaries to shape and constrain meaning, but are not rigidly fixed and determinate” (p. 12).

The difference in the ideas presented by Dewey and Bereiter lies in the problem design. For Dewey, scientific knowledge could be learnt from inquiring into practical problems that learners are familiar with. After all, Dewey viewed “modern science as continuous with, and to some degree an outgrowth and refinement of, practical or ‘common-sense’ inquiry” (Brown, 2012 ). For Bereiter, he acknowledged the importance of familiar experiences, but instead of using them as starting points for learning science, he argued that practical problems are limiting in helping learners acquire higher-order knowledge. Instead, he advocated for learners to organize their knowledge around problems that are complex, persistent and extended and requiring explanations to better understand the problems. Learners are to have a sense of the kinds of problems to which the specific concept is relevant before they can be said to have grasp the concept in a functionally useful way.

To connect between problem solving, scientific knowledge and everyday experiences, we need to examine ways to re-negotiate the disciplinary boundaries (such as epistemic understanding, object of inquiry, degree of precision) of science and make relevant connections to common-sense inquiry and to the problem at hand. Integrated STEM appears to be one way in which the disciplinary boundaries of science can be re-negotiated to include practices from the fields of technology, engineering and mathematics. In integrated STEM learning, inquiry is seen more holistically as a fluid process in which the outcomes are not absolute but are tentative. The fluidity of the inquiry process is reflected in the non-deterministic inquiry approach. This means that students can use science inquiry, engineering design, design process or any other inquiry approaches that fit to arrive at the solution. This hybridity of inquiry between science, common-sense and problems allows for some familiar aspects of the science inquiry process to be applied to understand and generate solutions to familiar everyday problems. In attempting to infuse elements of common-sense inquiry with science inquiry in problem-solving, logic plays an important role to help learners make connections. Hypothetically, we argue that with increasing exposure to less familiar ways of thinking such as those associated with science inquiry, students’ familiarity with scientific reasoning increases, and hence such ways of thinking gradually become part of their common-sense, which students could employ to solve future relevant problems. The theoretical ideas related to complexities of problems, the different forms of inquiry afforded by different problems and the arguments for engaging in problem solving motivated us to examine empirically how learners engage with ill-structured problems to generate problem-centred knowledge. Of particular interest to us is how learners and teachers weave between practical and scientific reasoning as they inquire to integrate the components in the original problem into a unified whole.

3.1 Context

The integrated STEM activity in our study was planned using the S-T-E-M quartet instructional framework (Tan et al., 2019 ). The S-T-E-M quartet instructional framework positions complex, persistent and extended problems at its core and focusses on the vertical disciplinary knowledge and understanding of the horizontal connections between the disciplines that could be gained by learners through solving the problem (Tan et al., 2019 ). Figure  1 depicts the disciplinary aspects of the problem that was presented to the students. The activity has science and engineering as the two lead disciplines. It spanned three 1-h lessons and required students to both learn and apply relevant scientific conceptual knowledge to solve a complex, real-world problem through processes that resemble the engineering design process (Wheeler et al., 2019 ).

Connections across disciplines in integrate STEM activity

Frequency of different types of reasoning

In the first session (1 h), students were introduced to the problem and its context. The problem pertains to the issue of limited farmland in a land scarce country that imports 90% of food (Singapore Food Agency [SFA], 2020 ). The students were required to devise a solution by applying knowledge of the conditions required for photosynthesis and plant growth to design and build a vertical farming system to help farmers increase crop yield with limited farmland. This context was motivated by the government’s effort to generate interests and knowledge in farming to achieve the 30 by 30 goal—supplying 30% of country’s nutritional needs by 2030. The scenario was a fictitious one where they were asked to produce 120 tonnes of Kailan (a type of leafy vegetable) with two hectares of land instead of the usual six hectares over a specific period. In addition to the abovementioned constraints, the teacher also discussed relevant success criteria for evaluating the solution with the students. Students then researched about existing urban farming approaches. They were given reading materials pertaining to urban farming to help them understand the affordances and constraints of existing solutions. In the second session (6 h), students engaged in ideation to generate potential solutions. They then designed, built and tested their solution and had opportunities to iteratively refine their solution. Students were given a list of materials (e.g. mounting board, straws, ice-cream stick, glue, etc.) that they could use to design their solutions. In the final session (1 h), students presented their solution and reflected on how well their solution met the success criteria. The prior scientific conceptual knowledge that students require to make sense of the problem include knowledge related to plant nutrition, namely, conditions for photosynthesis, nutritional requirements of Kailin and growth cycle of Kailin. The problem resembles a real-world problem that requires students to engage in some level of explanation of their design solution.

A total of 113 eighth graders (62 boys and 51 girls), 14-year-olds, from six classes and their teachers participated in the study. The students and their teachers were recruited as part of a larger study that examined the learning experiences of students when they work on integrated STEM activities that either begin with a problem, a solution or are focused on the content. Invitations were sent to schools across the country and interested schools opted in for the study. For the study reported here, all students and teachers were from six classes within a school. The teachers had all undergone 3 h of professional development with one of the authors on ways of implementing the integrated STEM activity used in this study. During the professional development session, the teachers learnt about the rationale of the activity, familiarize themselves with the materials and clarified the intentions and goals of the activity. The students were mostly grouped in groups of three, although a handful of students chose to work independently. The group size of students was not critical for the analysis of talk in this study as the analytic focus was on the kinds of knowledge applied rather than collaborative or group think. We assumed that the types of inquiry adopted by teachers and students were largely dependent on the nature of problem. Eighth graders were chosen for this study since lower secondary science offered at this grade level is thematic and integrated across biology, chemistry and physics. Furthermore, the topic of photosynthesis is taught under the theme of Interactions at eighth grade (CPDD, 2021 ). This thematic and integrated nature of science at eighth grade offered an ideal context and platform for integrated STEM activities to be trialled.

The final lessons in a series of three lessons in each of the six classes was analysed and reported in this study. Lessons where students worked on their solutions were not analysed because the recordings had poor audibility due to masking and physical distancing requirements as per COVID-19 regulations. At the start of the first lesson, the instructions given by the teacher were:

You are going to present your models. Remember the scenario that you were given at the beginning that you were tasked to solve using your model. …. In your presentation, you have to present your prototype and its features, what is so good about your prototype, how it addresses the problem and how it saves costs and space. So, this is what you can talk about during your presentation. ….. pay attention to the presentation and write down questions you like to ask the groups after the presentation… you can also critique their model, you can evaluate, critique and ask questions…. Some examples of questions you can ask the groups are? Do you think your prototype can achieve optimal plant growth? You can also ask questions specific to their models.

3.2 Data collection

Parental consent was sought a month before the start of data collection. The informed consent adhered to confidentiality and ethics guidelines as described by the Institutional Review Board. The data collection took place over a period of one month with weekly video recording. Two video cameras, one at the front and one at the back of the science laboratory were set up. The front camera captured the students seated at the front while the back video camera recorded the teacher as well as the groups of students at the back of the laboratory. The video recordings were synchronized so that the events captured from each camera can be interpreted from different angles. After transcription of the raw video files, the identities of students were substituted with pseudonyms.

3.3 Data analysis

The video recordings were analysed using the qualitative content analysis approach. Qualitative content analysis allows for patterns or themes and meanings to emerge from the process of systematic classification (Hsieh & Shannon, 2005 ). Qualitative content analysis is an appropriate analytic method for this study as it allows us to systematically identify episodes of practical inquiry and science inquiry to map them to the purposes and outcomes of these episodes as each lesson unfolds.

In total, six h of video recordings where students presented their ideas while the teachers served as facilitator and mentor were analysed. The video recordings were transcribed, and the transcripts were analysed using the NVivo software. Our unit of analysis is a single turn of talk (one utterance). We have chosen to use utterances as proxy indicators of reasoning practices based on the assumption that an utterance relates to both grammar and context. An utterance is a speech act that reveals both meaning and intentions of the speaker within specific contexts (Li, 2008 ).

Our research analytical lens is also interpretative in nature and the validity of our interpretation is through inter-rater discussion and agreement. Each utterance at the speaker level in transcripts was examined and coded either as relevant to practical reasoning or scientific reasoning based on the content. The utterances could be a comment by the teacher, a question by a student or a response by another student. Deductive coding is deployed with the two codes, practical reasoning and scientific reasoning derived from the theoretical ideas of Dewey and Bereiter as described earlier. Practical reasoning refers to utterances that reflect commonsensical knowledge or application of everyday understanding. Scientific reasoning refers to utterances that consist of scientifically oriented questions, scientific terms, or the use of empirical evidence to explain. Examples of each type of reasoning are highlighted in the following section. Each coded utterance is then reviewed for detailed description of the events that took place that led to that specific utterance. The description of the context leading to the utterance is considered an episode. The episodes and codes were discussed and agreed upon by two of the authors. Two coders simultaneously watched the videos to identify and code the episodes. The coders interpreted the content of each utterance, examine the context where the utterance was made and deduced the purpose of the utterance. Once each coder has established the sense-making aspect of the utterance in relation to the context, a code of either practical reasoning or scientific reasoning is assigned. Once that was completed, the two coders compared their coding for similarities and differences. They discussed the differences until an agreement was reached. Through this process, an agreement of 85% was reached between the coders. Where disagreement persisted, codes of the more experienced coder were adopted.

4 Results and Discussion

The specific STEM lessons analysed were taken from the lessons whereby students presented the model of their solutions to the class for peer evaluation. Every group of students stood in front of the class and placed their model on the bench as they presented. There was also a board where they could sketch or write their explanations should they want to. The instructions given by the teacher to the students were to explain their models and state reasons for their design.

4.1 Prevalence of Reasoning

The 6h of videos consists of 1422 turns of talk. Three hundred four turns of talk (21%) were identified as talk related to reasoning, either practical reasoning or scientific reasoning. Practical reasoning made up 62% of the reasoning turns while 38% were scientific reasoning (Fig. 2 ).

The two types of reasoning differ in the justifications that are used to substantiate the claims or decisions made. Table 1 describes the differences between the two categories of reasoning.

4.2 Applications of Scientific Reasoning

Instances of engagement with scientific reasoning (for instance, using scientific concepts to justify, raising scientifically oriented questions, or providing scientific explanations) revolved around the conditions for photosynthesis and the concept of energy conversion when students were presenting their ideas or when they were questioned by their peers. For example, in explaining the reason for including fish in their plant system, one group of students made connection to cyclical energy transfer: “…so as the roots of the plants submerged in the water, faeces from the fish will be used as fertilizers so that the plant can grow”. The students considered how organic matter that is still trapped within waste materials can be released and taken up by plants to enhance the growth. The application of scientific reasoning made their design one that is innovative and sustainable as evaluated by the teacher. Some students attempted more ecofriendly designs by considering energy efficiencies through incorporating water turbines in their farming systems. They applied the concept of different forms of energy and energy conversion when their peers inquired about their design. The same scientific concepts were explained at different levels of details by different students. At one level, the students explained in a purely descriptive manner of what happens to the different entities in their prototypes, with implied changes to the forms of energy─ “…spins then generates electricity. So right, when the water falls down, then it will spin. The water will fall on the fan blade thing, then it will spin and then it generates electricity. So, it saves electricity, and also saves water”. At another level, students defended their design through an explanation of energy conversion─ “…because when the water flows right, it will convert gravitational potential energy so, when it reaches the bottom, there is not really much gravitational potential energy”. While these instances of applying scientific reasoning indicated that students have knowledge about the scientific phenomena and can apply them to assist in the problem-solving process, we are not able to establish if students understood the science behind how the dynamo works to generate electricity. Students in eighth grade only need to know how a generator works at a descriptive level and the specialized understanding how a dynamo works is beyond the intended learning outcomes at this grade level.

The application of scientific concepts for justification may not always be accurate. For instance, the naïve conception that students have about plants only respiring at night and not in the day surfaced when one group of students tried to justify the growth rates of Kailan─ “…I mean, they cannot be making food 24/7 and growing 24/7. They have nighttime for a reason. They need to respire”. These students do not appreciate that plants respire in the day as well, and hence respiration occurs 24/7. This naïve conception that plants only respire at night is one that is common among learners of biology (e.g. Svandova, 2014 ) since students learn that plant gives off oxygen in the day and takes in oxygen at night. The hasty conclusion to that observation is that plants carry out photosynthesis in the day and respire at night. The relative rates of photosynthesis and respiration were not considered by many students.

Besides naïve conceptions, engagement with scientific ideas to solve a practical problem offers opportunities for unusual and alternative ideas about science to surface. For instance, another group of students explained that they lined up their plants so that “they can take turns to absorb sunlight for photosynthesis”. These students appear to be explaining that the sun will move and depending on the position of the sun, some plants may be under shade, and hence rates of photosynthesis are dependent on the position of the sun. However, this idea could also be interpreted as (1) the students failed to appreciate that sunlight is everywhere, and (2) plants, unlike animals, particularly humans, do not have the concept of turn-taking. These diverse ideas held by students surfaced when students were given opportunities to apply their knowledge of photosynthesis to solve a problem.

4.3 Applications of Practical Reasoning

Teachers and students used more practical reasoning during an integrated STEM activity requiring both science and engineering practices as seen from 62% occurrence of practical reasoning compared with 38% for scientific reasoning. The intention of the activity to integrate students’ scientific knowledge related to plant nutrition to engineering practice of building a model of vertical farming system could be the reason for the prevalence of practical reasoning. The practical reasoning used related to structural design considerations of the farming system such as how water, light and harvesting can be carried out in the most efficient manner. Students defended the strengths of designs using logic based on their everyday experiences. In the excerpt below (transcribed verbatim), we see students applied their everyday experiences when something is “thinner” (likely to mean narrower), logically it would save space. Further, to reach a higher level, you use a machine to climb up.

Excerpt 1. “Thinner, more space” Because it is more thinner, so like in terms of space, it’s very convenient. So right, because there is – because it rotates right, so there is this button where you can stop it. Then I also installed steps, so that – because there are certain places you can’t reach even if you stop the – if you stop the machine, so when you stop it and you climb up, and then you see the condition of the plants, even though it costs a lot of labour, there is a need to have an experienced person who can grow plants. Then also, when like – when water reach the plants, cos the plants I want to use is soil-based, so as the water reach the soil, the soil will xxx, so like the water will be used, and then we got like – and then there’s like this filter that will filter like the dirt.

In the examples of practical reasoning, we were not able to identify instances where students and teachers engaged with discussion around trade-off and optimisation. Understanding constraints, trade-offs and optimisations are important ideas in informed design matrix for engineering as suggested by Crismond and Adams ( 2012 ). For instance, utterances such as “everything will be reused”, “we will be saving space”, “it looks very flimsy” or “so that it can contains [sic] the plants” were used. These utterances were made both by students while justifying their own prototypes and also by peers who challenged the design of others. Longer responses involving practical reasoning were made based on common-sense, everyday logic─ “…the product does not require much manpower, so other than one or two supervisors like I said just now, to harvest the Kailan, hence, not too many people need to be used, need to be hired to help supervise the equipment and to supervise the growth”. We infer that the higher instances of utterances related to practical reasoning could be due to the presence of more concrete artefacts that is shown, and the students and teachers were more focused on questioning the structure at hand. This inference was made as instructions given by the teacher at the start of students’ presentation focus largely on the model rather than the scientific concepts or reasoning behind the model.

4.4 Intersection Between Scientific and Practical Reasoning

Comparing science subject matter knowledge and problem-solving to the idea of categories and placement (Buchanan, 1992 ), subject matter is analogous to categories where meanings are fixed with well-established epistemic practices and norms. The problem-solving process and design of solutions are likened to placements where boundaries are less rigid, hence opening opportunities for students’ personal experiences and ideas to be presented. Placements allow students to apply their knowledge from daily experiences and common-sense logic to justify decisions. Common-sense knowledge and logic are more accessible, and hence we observe higher frequency of usage. Comparatively, while science subject matter (categories) is also used, it is observed less frequently. This could possibly be due either to less familiarity with the subject matter or lack of appropriate opportunity to apply in practical problem solving. The challenge for teachers during implementation of a STEM problem-solving activity, therefore, lies in the balance of the application of scientific and practical reasoning to deepen understanding of disciplinary knowledge in the context of solving a problem in a meaningful manner.

Our observations suggest that engaging students with practical inquiry tasks with some engineering demands such as the design of modern farm systems offers opportunities for them to convert their personal lived experiences into feasible concrete ideas that they can share in a public space for critique. The peer critique following the sharing of their practical ideas allows for both practical and scientific questions to be asked and for students to defend their ideas. For instance, after one group of students presented their prototype that has silvered surfaces, a student asked a question: “what is the function of the silver panels?”, to which his peers replied : “Makes the light bounce. Bounce the sunlight away and then to other parts of the tray.” This question indicated that students applied their knowledge that shiny silvered surfaces reflect light, and they used this knowledge to disperse the light to other trays where the crops were growing. An example of a practical question asked was “what is the purpose of the ladder?”, to which the students replied: “To take the plants – to refill the plants, the workers must climb up”. While the process of presentation and peer critique mimic peer review in the science inquiry process, the conceptual knowledge of science may not always be evident as students paid more attention to the design constraints such as lighting, watering, and space that was set in the activity. Given the context of growing plants, engagement with the science behind nutritional requirements of plants, the process of photosynthesis, and the adaptations of plants could be more deliberately explored.

5 Conclusion

The goal of our work lies in applying the theoretical ideas of Dewey and Bereiter to better understand reasoning practices in integrate STEM problem solving. We argue that this is a worthy pursue to better understand the roles of scientific reasoning in practical problem solving. One of the goals of integrated STEM education in schools is to enculture students into the practices of science, engineering and mathematics that include disciplinary conceptual knowledge, epistemic practices, and social norms (Kelly & Licona, 2018 ). In the integrated form, the boundaries and approaches to STEM learning are more diverse compared with monodisciplinary ways of problem solving. For instance, in integrated STEM problem solving, besides scientific investigations and explanations, students are also required to understand constraints, design optimal solutions within specific parameters and even to construct prototypes. For students to learn the ways of speaking, doing and being as they participate in integrated STEM problem solving in schools in a meaningful manner, students could benefit from these experiences.

With reference to the first research question of What is the extent of practical and scientific reasoning in integrated STEM problem solving, our analysis suggests that there are fewer instances of scientific reasoning compared with practical reasoning. Considering the intention of integrated STEM learning and adopting Bereiter’s idea that students should learn higher-order conceptual knowledge through engagement with problem solving, we argue for a need for scientific reasoning to be featured more strongly in integrated STEM lessons so that students can gain higher order scientific conceptual knowledge. While the lessons observed were strong in design and building, what was missing in generating solutions was the engagement in investigations, where learners collected or are presented with data and make decisions about the data to allow them to assess how viable the solutions are. Integrated STEM problems can be designed so that science inquiry can be infused, such as carrying out investigations to figure out relationships between variables. Duschl and Bybee ( 2014 ) have argued for the need to engage students in problematising science inquiry and making choices about what works and what does not.

With reference to the second research question , What is achieved through practical and scientific reasoning during integrated STEM problem solving? , our analyses suggest that utterance for practical reasoning are typically used to justify the physical design of the prototype. These utterances rely largely on what is observable and are associated with basic-level knowledge and experiences. The higher frequency of utterances related to practical reasoning and the nature of the utterances suggests that engagement with practical reasoning is more accessible since they relate more to students’ lived experiences and common-sense. Bereiter ( 1992 ) has urged educators to engage learners in learning that is beyond basic-level knowledge since accumulation of basic-level knowledge does not lead to higher-level conceptual learning. Students should be encouraged to use scientific knowledge also to justify their prototype design and to apply scientific evidence and logic to support their ideas. Engagement with scientific reasoning is preferred as conceptual knowledge, epistemic practices and social norms of science are more widely recognised compared with practical reasoning that are likely to be more varied since they rely on personal experiences and common-sense. This leads us to assert that both context and content are important in integrated STEM learning. Understanding the context or the solution without understanding the scientific principles that makes it work makes the learning less meaningful since we “…cannot strip learning of its context, nor study it in a ‘neutral’ context. It is always situated, always relayed to some ongoing enterprise”. (Bruner, 2004 , p. 20).

To further this discussion on how integrated STEM learning experiences harness the ideas of practical and scientific reasoning to move learners from basic-level knowledge to higher-order conceptual knowledge, we propose the need for further studies that involve working with teachers to identify and create relevant problems-of-explanations that focuses on feasible, worthy inquiry ideas such as those related to specific aspects of transportation, alternative energy sources and clean water that have impact on the local community. The design of these problems can incorporate opportunities for systematic scientific investigations and scaffolded such that there are opportunities to engage in epistemic practices of the constitute disciplines of STEM. Researchers could then examine the impact of problems-of-explanations on students’ learning of higher order scientific concepts. During the problem-solving process, more attention can be given to elicit students’ initial and unfolding ideas (practical) and use them as a basis to start the science inquiry process. Researchers can examine how to encourage discussions that focus on making meaning of scientific phenomena that are embedded within specific problems. This will help students to appreciate how data can be used as evidence to support scientific explanations as well as justifications for the solutions to problems. With evidence, learners can be guided to work on reasoning the phenomena with explanatory models. These aspects should move engagement in integrated STEM problem solving from being purely practice to one that is explanatory.

6 Limitations

There are four key limitations of our study. Firstly, the degree of generalisation of our observations is limited. This study sets out to illustrate what how Dewey and Bereiter’s ideas can be used as lens to examine knowledge used in problem-solving. As such, the findings that we report here is limited in its ability to generalise across different contexts and problems. Secondly, the lessons that were analysed came from teacher-frontal teaching and group presentation of solution and excluded students’ group discussions. We acknowledge that there could potentially be talk that could involve practical and scientific reasonings within group work. There are two practical consideration for choosing to analyse the first and presentation segments of the suite of lesson. Firstly, these two lessons involved participation from everyone in class and we wanted to survey the use of practical and scientific reasoning by the students as a class. Secondly, methodologically, clarity of utterances is important for accurate analysis and as students were wearing face masks during the data collection, their utterances during group discussions lack the clarity for accurate transcription and analysis. Thirdly, insights from this study were gleaned from a small sample of six classes of students. Further work could involve more classes of students although that could require more resources devoted to analysis of the videos. Finally, the number of students varied across groups and this could potentially affect the reasoning practices during discussions.

Data Availability

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Aikenhead, G. S. (2006). Science education for everyday life: Evidence-based practice . Teachers College Press.

Google Scholar  

Bereiter, C. (1992). Referent-centred and problem-centred knowledge: Elements of an educational epistemology. Interchange, 23 (4), 337–361.

Article   Google Scholar  

Breiner, J. M., Johnson, C. C., Harkness, S. S., & Koehler, C. M. (2012). What is STEM? A discussion about conceptions of STEM in education and partnership. School Science and Mathematics, 112 (1), 3–11. https://doi.org/10.1111/j.194908594.2011.00109.x

Brown, M. J. (2012). John Dewey’s logic of science. HOPS: The Journal of the International Society for the History of Philosophy of Science, 2 (2), 258–306.

Bruner, J. (2004). The psychology of learning: A short history (pp.13–20). Winter: Daedalus.

Bryan, L. A., Moore, T. J., Johnson, C. C., & Roehrig, G. H. (2016). Integrated STEM education. In C. C. Johnson, E. E. Peters-Burton, & T. J. Moore (Eds.), STEM road map: A framework for integrated STEM education (pp. 23–37). Routledge.

Buchanan, R. (1992). Wicked problems in design thinking. Design Issues, 8 (2), 5–21.

Bybee, R. W. (2013). The case for STEM education: Challenges and opportunities . NSTA Press.

Crismond, D. P., & Adams, R. S. (2012). The informed design teaching and learning matrix. Journal of Engineering Education, 101 (4), 738–797.

Cunningham, C. M., & Lachapelle, P. (2016). Experiences to engage all students. Educational Designer , 3(9), 1–26. https://www.educationaldesigner.org/ed/volume3/issue9/article31/

Curriculum Planning and Development Division [CPDD] (2021). 2021 Lower secondary science express/ normal (academic) teaching and learning syllabus . Singapore: Ministry of Education.

Delahunty, T., Seery, N., & Lynch, R. (2020). Exploring problem conceptualization and performance in STEM problem solving contexts. Instructional Science, 48 , 395–425. https://doi.org/10.1007/s11251-020-09515-4

Dewey, J. (1938). Logic: The theory of inquiry . Henry Holt and Company Inc.

Dewey, J. (1910a). Science as subject-matter and as method. Science, 31 (787), 121–127.

Dewey, J. (1910b). How we think . D.C. Heath & Co Publishers.

Book   Google Scholar  

Duschl, R. A., & Bybee, R. W. (2014). Planning and carrying out investigations: an entry to learning and to teacher professional development around NGSS science and engineering practices. International Journal of STEM Education, 1 (12). DOI: https://doi.org/10.1186/s40594-014-0012-6 .

Gale, J., Alemder, M., Lingle, J., & Newton, S (2000). Exploring critical components of an integrated STEM curriculum: An application of the innovation implementation framework. International Journal of STEM Education, 7(5), https://doi.org/10.1186/s40594-020-0204-1 .

Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15 (9), 1277–1288.

Jonassen, D. H. (2000). Toward a design theory of problem solving. ETR&D, 48 (4), 63–85.

Kelly, G., & Licona, P. (2018). Epistemic practices and science education. In M. R. Matthews (Ed.), History, philosophy and science teaching: New perspectives (pp. 139–165). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-62616-1 .

Lee, O., & Luykx, A. (2006). Science education and student diversity: Synthesis and research agenda . Cambridge University Press.

Li, D. (2008). The pragmatic construction of word meaning in utterances. Journal of Chinese Language and Computing, 18 (3), 121–137.

National Research Council. (1996). The National Science Education standards . National Academy Press.

National Research Council (2000). Inquiry and the national science education standards: A guide for teaching and learning. Washington, DC: The National Academies Press. https://doi.org/10.17226/9596 .

OECD (2018). The future of education and skills: Education 2030. Downloaded on October 3, 2020 from https://www.oecd.org/education/2030/E2030%20Position%20Paper%20(05.04.2018).pdf

Park, W., Wu, J.-Y., & Erduran, S. (2020) The nature of STEM disciplines in science education standards documents from the USA, Korea and Taiwan: Focusing on disciplinary aims, values and practices.  Science & Education, 29 , 899–927.

Pleasants, J. (2020). Inquiring into the nature of STEM problems: Implications for pre-college education. Science & Education, 29 , 831–855.

Roehrig, G. H., Dare, E. A., Ring-Whalen, E., & Wieselmann, J. R. (2021). Understanding coherence and integration in integrated STEM curriculum. International Journal of STEM Education, 8(2), https://doi.org/10.1186/s40594-020-00259-8

SFA (2020). The food we eat . Downloaded on May 5, 2021 from https://www.sfa.gov.sg/food-farming/singapore-food-supply/the-food-we-eat

Svandova, K. (2014). Secondary school students’ misconceptions about photosynthesis and plant respiration: Preliminary results. Eurasia Journal of Mathematics, Science, & Technology Education, 10 (1), 59–67.

Tan, M. (2020). Context matters in science education. Cultural Studies of Science Education . https://doi.org/10.1007/s11422-020-09971-x

Tan, A.-L., Teo, T. W., Choy, B. H., & Ong, Y. S. (2019). The S-T-E-M Quartet. Innovation and Education , 1 (1), 3. https://doi.org/10.1186/s42862-019-0005-x

Wheeler, L. B., Navy, S. L., Maeng, J. L., & Whitworth, B. A. (2019). Development and validation of the Classroom Observation Protocol for Engineering Design (COPED). Journal of Research in Science Teaching, 56 (9), 1285–1305.

World Economic Forum (2020). Schools of the future: Defining new models of education for the fourth industrial revolution. Retrieved on Jan 18, 2020 from https://www.weforum.org/reports/schools-of-the-future-defining-new-models-of-education-for-the-fourth-industrial-revolution/

Download references

Acknowledgements

The authors would like to acknowledge the contributions of the other members of the research team who gave their comment and feedback in the conceptualization stage.

This study is funded by Office of Education Research grant OER 24/19 TAL.

Author information

Authors and affiliations.

Natural Sciences and Science Education, meriSTEM@NIE, National Institute of Education, Nanyang Technological University, Singapore, Singapore

Aik-Ling Tan, Yann Shiou Ong, Yong Sim Ng & Jared Hong Jie Tan

You can also search for this author in PubMed   Google Scholar

Contributions

The first author conceptualized, researched, read, analysed and wrote the article.

The second author worked on compiling the essential features and the variations tables.

The third and fourth authors worked with the first author on the ideas and refinements of the idea.

Corresponding author

Correspondence to Yann Shiou Ong .

Ethics declarations

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Tan, AL., Ong, Y.S., Ng, Y.S. et al. STEM Problem Solving: Inquiry, Concepts, and Reasoning. Sci & Educ 32 , 381–397 (2023). https://doi.org/10.1007/s11191-021-00310-2

Download citation

Accepted : 28 November 2021

Published : 29 January 2022

Issue Date : April 2023

DOI : https://doi.org/10.1007/s11191-021-00310-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Practical Inquiry
• Science Inquiry
• Referent-centered knowledge
• Problem-centered knowledge
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 05 February 2018

The role of problem solving ability on innovative behavior and opportunity recognition in university students

• Ji Young Kim 1 ,
• Dae Soo Choi 1 ,
• Chang-Soo Sung 1 &
• Joo Y. Park 2  

Journal of Open Innovation: Technology, Market, and Complexity volume  4 , Article number:  4 ( 2018 ) Cite this article

25k Accesses

31 Citations

1 Altmetric

Metrics details

Universities engage in entrepreneurship education to increase social value creation, through students’ new opportunities recognition. However, there are not enough of empirical researches on whether the current entrepreneurship education can be differentiated from other curriculum to improve the opportunity recognition process. This study argues that it is very important for cognitive abilities to be manifested as behavior when students in university are new opportunities recognition. For this purpose, the relationship between problem solving ability, innovation behavior, and opportunity perception was verified empirically. This study was conducted on 203 students who took entrepreneurship education courses at Korean universities. The results of this study showed that problem solving ability positively influenced innovation behavior and opportunity perception. Innovation behavior was identified as a key parameter that partially mediated the relationship between problem solving ability and innovation behavior. The implication of this study is to prove the relationship between individual ‘s problem - solving ability considering the characteristics of education in Korea and the opportunity through innovative behavior and various learning strategies to help entrepreneurship education to design better courses for the future It has important implications for strategic pedagogy that can enhance behavioral elements in development.

It is the new opportunity recognition that all firms focus on for a new economic paradigm (Ancona and Caldwell, 1992 ). Recognizing high opportunities can significantly improve profit, growth, and / or competitive positioning. And this new opportunity leads to innovation. From a conceptual point of view, research is continuing on the question of ‘what is opportunity’ and ‘where is opportunity’ (Gartner and Carter, 2003 ; Venkataraman & Sarasvathy, 2001 ). Research on the discovery and realization of new opportunities is a very important research area that suggests how to discover and utilize creative opportunities that create new value and profit for pre-service workers, and is the ultimate goal of entrepreneurship education. (Kim et al., 2016 ). Particularly, there is a lot of debate about the relationship between opportunity perception and personal characteristics. Despite many arguments, however, research on individual characteristics and opportunity perceptions is still insufficient, and a unified opinion has not been created due to differences between cognitive and behavioral theories (Ko & Butler, 2003 ). In particular, there is much controversy over the relationship between opportunity recognition and personal traits, and research has been continuing to demonstrate that organizational learning in organizations can influence opportunity recognition (Shane & Venkataraman, 2000 ). In particular, learning enhances cognitive ability, which is an opportunity that leads to opportunity recognition through the manifestation of behavior (Lumpkin and Dess, 2004 ). Many studies have also demonstrated the difference in behavior that successful entrepreneurs see as contributing to their ability to recognize opportunities and create innovative business ideas (Dyer et al., 2008 ; Kim et al., 2017 ). For example, Alvarez and Barney ( 2005 ) argue for mountain climbing and mountain building to understand the implications of entrepreneurial behavior in relation to these theories. In other words, a new opportunity for entrepreneurs is not a passive case that is generally found and climbed by climbers such as mountains, but rather by the actions of entrepreneurs, creating competition for the market, creating another market, Is the same. Therefore, in order for a person’s cognitive ability to recognize a new opportunity, it must focus on manifesting an action that can realize an innovative idea. In this regard, Kanter ( 1988 ) proved the relationship between new opportunity recognition and those with innovative tendencies and regarded this new opportunity recognition as innovation activity through organizational education. Scott and Bruce ( 1994 ) have integrated a number of research flows into innovation pioneers to develop and test individual innovative behavioral models. In particular, they argued that individual problem-solving styles are very important to induce innovative behavior. Although there are a number of studies on problem solving ability, innovation behavior, and new opportunities, most of the opportunistic researches have been conducted in organizational units of companies. Is still insufficient. Furthermore, unified opinions were not created due to differences between cognitive theory and behavioral theory (Ko & Butler, 2003 ). It is also true that the effects of entrepreneurship education in university have not been studied empirically because they are mainly focused on promoting cognitive ability and applied to various kinds of teaching methods.

This study argues that it is very important for cognitive abilities to be manifested as behavior that. “Through” courses, In other words, it is very important to induce students to act through ‘learning through process’ learning through behavioral learning by providing students with some (virtual or real) business to start doing some of the actions of the entrepreneur. When students in university are new opportunity recognition. Especially, entrepreneurship education, which ultimately focuses on whether it is a new opportunity, is very important to induce behavior through behavior learning beyond the cognitive ability as the general education curriculum. Particularly, innovative behaviors that create and realize innovative ideas are very important for new opportunity recognition (Paine & Organ, 2000 ).In order to achieve this, various kinds of teaching methods are being pursued in the university, but studies on the effectiveness of behavioral learning have not been studied yet. In this study, we are based on team-based learning among various teaching methods for behavior learning that leads to innovative behaviors. Team learning instructional activity sequence designed by Michaelsen and Sweet ( 2008 ), the most well known team-based learning in entrepreneurship education as in class-primarily group work and outside class-primarily individual work. In this way, we demonstrate empirically the relationship between individual problem solving ability and opportunity through innovative behavior, and develop a variety of learning strategies that help entrepreneurship education to design better courses for the future. I would like to point out some implications for strategic pedagogy to increase the element.

The paper proceeds as follows: Initially we present the theory of innovative behavior with individual problem-solving ability, innovative behavior and opportunity recognition. We develop hypotheses to confirm its basic predictions in the student context. Finally, we link the findings with the wider social effect of entrepreneurship literature and highlight the theoretical contributions and practical implications.

Theoretical background

‘opportunity recognition’ as entrepreneurship education unit of analysis.

A commonly focused analysis in entrepreneurship research over the last 30 years has been the ‘opportunity’, most simply defined as any situation in which new products or services can be development of production (Casson, 1982 ; Shane & Venkataraman, 2000 ; Venkataraman, 1997 ). The definition of opportunity recognition is defined in many ways, but opportunity is defined as a perceived means of generating economic value (ie, profit) that has not been exploited previously and is not currently exploited by others. If opportunity is defined in this way, opportunity recognition can be defined as a cognitive process (or process) that concludes that an individual has identified an opportunity (Baron and Ensley, 2006 ). Kirzner ( 1997 ) pointed out that the distribution of information in society affects the discovery of entrepreneurial opportunities and that only a few individuals can identify and recognize specific opportunities in the market. The process of finding opportunities also depends on the individual’s ability and discovery (Stevenson & Gumpert, 1985 ). For example, people may miss opportunities due to a lack of cognitive ability to change external environments (Stevenson & Gumpert, 1985 ). Only those who recognize and value the existence of opportunity can benefit from new opportunities (Ardichvili et al., 2003a , b ; Shane & Venkataraman, 2000 ). Opportunity recognition is an early step in transforming value into a business concept that creates value and generates revenue and distinguishes it from the aggressive stages of detailed assessment and development of recognized opportunities and potential economic value. The focus of the new venture business is also an innovative opportunity to create new opportunities rather than merely expanding or repeating existing business models (Gaglio & Katz, 2001 ). As a result, universities need to make use of a variety of initiatives to educate students to recognize innovative opportunities. Therefore, entrepreneurship education aimed at a new opportunity recognition should be able to provide learning opportunities based on various theories of favorable conditions for new business creation and the types of traits required for new ventures (Garavan & O’Cinne’ide, 1994 ).

Based on these considerations, we also define opportunity recognition as the formation of beliefs that can be translated into actions in order to understand the signals of change (new information on new conditions) and respond to these changes.

Problem-solving ability and innovative behavior of education for students

Problem-solving abilities have been proven to be one of the key factors for success in organizations and personal careers (Anderson & Anderson 1995 ). Through decades of research data, organizations and schools have studied factors that affect improvement. Problem-solving abilities are defined in a number of prior studies, and problem-solving abilities in a volatile and sophisticated knowledge- and technology-based industry are an important ability to drive innovation and sustainable growth and development in the industry. Table  1 show the concept of problem solving ability defined in previous research.

There have been a number of previous studies, emphasis has been placed on the importance and meaning of rational problem-solving processes in order to improve problem-solving abilities, and research has focused on individual problem solving styles (Woodman et al., 1993 ; Scott & Bruce, 1994 ). According to the personal innovation behavior model of Scott and Bruce ( 1994 ), climate has shown individual innovative behavior as a result of individuals signaling the organization’s expectations of behavior and the potential consequences of action. Innovative organizations are, last but not least, equipment, facilities and time, including the direction of creativity and innovative change (Kanter, 1983 ; Siegel & Kaemmerer, 1978 ) Proper supply of such resources is important to innovation (Amabile, 1988 ; Van de Ven & Angle, 1989 ; Dubickis & Gaile-Sarkane, 2017 ). Based on a study of Koestler’s ( 1964 ) creative thinking, Jabri conceptualized a problem-solving style consisting of two independent thinking styles. He uses a structured problem-solving styles that is based on associative thinking, follows a set of rules, resolves reasonably logically, and uses an intuitive problem-solving ability that focuses on problem-solving, not tied to existing rules with multiple ideas. Intuitive problem solving styles tend to process information from different paradigms simultaneously. It is therefore more likely to create new problem solutions as possible (Isaksen, 1987 ; Kirton, 1976 ). However, style assessment is not desirable because the style of problem solving affects style differently depending on the individual problem-solving situations (Scott & Bruce, 1994 ). We are proposing a role for the University to encourage innovative behavior based on the individuality of our students in order to recognize new opportunities through education about Scott and Bruce’s innovative behavioral models and diverse entrepreneurship education approaches. And involvement of resources, such as entrepreneurship awareness programs, ultimately leads to the identification of individual characteristics and innovation. In addition, current Korean entrepreneurship education is mainly focused on cognitive learning to improve problem solving ability, and one aspect of cognitive learning plays an important role in learning process of new venture firms. This study has a more direct focus on behavior learning such as team-based learning.

Hypothesis development

Problem-solving ability and innovative behavior.

Problem solving is to discover knowledge and skills that reach the target country by interfering with a set of processes and goals where the solution is unknown, unfamiliar, or reaching a new state of goal (Jonassen, 2004 ; Inkinen, 2015 ). There are various approaches to solve this problem. To solve problems and improve problem solving with a successful solution experience, you should adopt the method that best suits your problem solution. You need to select the appropriate inputs for the solution elements and a flexible process structure. Problem solving ability has been recognized as a key element of innovative behavior in responding to rapid changes with the ability to find various alternatives and predict outcomes from these alternatives to maximize positive results, minimize negative consequences, and select solutions to problems (Barron & Harrington, 1981 ; Jabri, 1991 ; Kirton, 1976 ). We pose the following hypotheses:

Hypothesis 1: Individual problem-solving ability has an effect on the innovative behavior of students.

Innovative behavior and opportunity recognition

Innovation involves introducing ideas from outside the organization, through creative processes, and linking these ideas to products or processes. Many scholars studying innovation recognize that designing ideas is only one step in the innovation process (Kanter, 1988 ). Innovation is changing at the organizational or individual level. Kanter, Scott and Bruce defined personal innovation. In other words, an innovation act starts with recognition of a problem, adoption of a new idea, or creation of a solution, and an individual with an innovative tendency wants to create a realistically realizable group with the sympathy of such an idea. Innovative individuals create prototypes for innovations that enable ideas to be realized specifically with goods or services and become productive use and social day merchandising. According to previous studies, opportunity perception can be seen as an individual’s corporate strategy that focuses on the perception and exploitation of individuals about potential business ideas and opportunities and finds resources to create innovative outcomes (Manev et al., 2005 ). New Venture Ideas (NVI) are imaginary combinations of product/service offerings; potential markets or users, and means of bringing these offerings into existence (Davidsson, 2015 ). From the viewpoint of a potential entrepreneur like a university student, entrepreneurship starts with an idea. This process continues with a range of practices including attractiveness and feasibility of an idea, gathering information to minimize value-related uncertainty and possibility and perhaps the main idea’s conformity ratio in terms of newly discovered needs (Hayton & Cholakova, 2012 ). Earlier we proposed that the program as a whole increases the students’ innovative behavior and that innovative performance is the new venture ideas. Since it is logical to assume that the relationship between innovative behavior and opportunity recognition. We pose the following hypotheses:

Hypothesis 2: Innovative behavior will be a more potent inducer of opportunity recognition.

Problem-solving ability and opportunity recognition

Among the many factors influencing opportunity perception, the problems that arise in the fourth industry, the knowledge-based industry of the twenty-first century, are unpredictable and unstructured; they cannot be solved with existing solutions and require creative problem-solving skills. In order to determine how to solve problem situations that are different from the current situation and have unknown results, problems are solved through the process of adjusting previous experience, knowledge, and intuition (Charles & Lester, 1982 ). Experience, knowledge, and intuition are applied simultaneously to a single problem, not individually or collectively, and the intellectual and creative results that can be quickly and effectively solved in problem solving are seen as problem solving abilities (Ardichvili et al., 2003a , b ). Empirical studies of problem-solving abilities and opportunity perceptions have provided strong evidence that there is a positive relationship between theoretical integrative processes and corporate opportunity recognition (Ucbasaran et al., 2009 ). Therefore, we hypothesized that:

Hypothesis 3: Problem solving ability has an effect on the opportunity recognition.

The respondents for this study were randomly selected from three universities in Korea. Most of the respondents in this study were Korean university students who experienced team-based learning during behavioral learning through entrepreneurship education. Since then, we have been guided by two main criteria when choosing these universities. First, students who take entrepreneurship courses are critical to their innovation behavior. This led us to realize that innovative behavior is an important factor in an individual’s survival and growth. The second is that the parallel process of theoretical and behavioral learning is highly satisfied. A pilot study was conducted to verify the reliability and validity of the research measurements with 28 students at a university. The results of the pilot study showed high clarity and reliability (Cronbach ‘s alphas were all above 0.70) ​​of the research measurements. The sample of the pilot study was not incorporated in the present study.

This study was conducted in a four - year undergraduate course (various majors) that took entrepreneurship courses in Korea university programs. Students in this course have a mix of students who have previously experienced entrepreneurship and those who have not. During the course, students were taught the theoretical lessons for 8 weeks and the team for the 8 weeks. The questionnaire was administered during the last week of the course.

The data were analyzed from 203 participants, out of a total of 209, of which 7 were not appropriate. Of the 203 participants, 27% were female and 73% were male and the grade distribution was 3% for freshmen, 12% for grade 2, 26% for grade 2, and 59% for grade 2. The main distribution is 26% in social science, 16% in business and economics, 39% in engineering, 11% in music and athletics and 7% in others (see Table  2 ).

Measurement

The structure of the model was measured by questionnaires (problem-solving ability, innovation behavior and opportunity recognition questionnaire) consisting of the scale taken from questionnaires verified in previous studies. Tool selection was performed on two criteria. First, the selected tool should measure the same structure (ie, the original measured structure had to be conceptually identical to the way the structure was defined in this study model). Secondly, the psychometric qualities of the instrument for the student had to be high.

Assessment of the factors was carried out through principal component analyses (varimax rotation with eigenvalues of 1.0 or above) of the scales connected to the same level of the model to confirm the uniqueness of the scales with respect to each other. This was supplemented by the computation of the internal consistency reliability of the scales (Cronbach’s α). These analyses were executed using the individual participants’ responses (Nunnally & Bernstein, 1994 ).

Problem- solving ability was measured on a 7-point Likert-scale (1 = ‘completely disagree’; 7 = ‘completely agree’). Jabri ( 1991 ) used a measurement tool to measure individual problem solving ability.

Innovative behavior was measured on a 7-point Likert-scale (1 = ‘completely disagree’; 7 = ‘completely agree’). In order to measure innovation behavior, we modified the questionnaire items to fit the intention of this study among the questionnaire items used by Scott and Bruce ( 1994 ) and Kim and Rho ( 2010 ).

Opportunity recognition was measured on a 7-point Likert-scale (1 = ‘completely disagree’; 7 = ‘completely agree’). In order to measure opportunity recognition, we modified the questionnaire items to fit the intention of this study among the questionnaire items used by Kim and Rho ( 2010 ).

Methods of analysis

The first two parts of the analysis were primarily based on (multiple) regression analyses. The last part of the analysis was informed through the path analyses. The adequacy of the models was assessed by AMOS 18(Arbuckle & Wothke, 2003 ). Models were all tested with standardized coefficients obtained from the Principal Component Analysis. To ascertain the model fit, we analyzed the comparative fit index (CFI), the normed fit index (NFI), the Root Mean Square Err of Approximation (RMSEA), the standardized root mean square residual (SRMR) and the chi-square test statistic.

Reliability and validity are essential psychometrics to be reported. The first step to evaluate those aspects was to use the Cronbach’s alpha and the composite reliability to test reliability of the proposed scales. The usual threshold level is 0.7 for newly developed measures (Fornell and Larcker, 1981 ). Values range from 0.69 to 0.79 in the case of Cronbach’s alpha, and from 0.85 to 0.92 in the case of composite reliability (see Table  3 ). Therefore, these scales may be considered as reliable. Next, we estimated the research model, displayed in Fig.  1 , using structural equation modeling (SEM) and AMOS 18 (Arbuckle & Wothke, 2003 ). Our analysis revealed an adequate measurement model with high factor loadings for all the items on the expected factors and communalities of each item exceeding 0.50. We discuss three fit indices that are generally considered as important (Hu & Bentler, 1998 ). First, the CFI-value represents the overall difference between observed and predicted correlations. A value of 0.04 which is situated well below the cut-off value of 0.08, suggests that the hypothesized model resembles the actual correlations. Secondly, Bentler’s CFI (comparative fit index) greater than 0.90 and 0.95 which is above the cut-off of 0.90 (Schumacker & Lomax, 1996 ). Thirdly, NFI greater greater than 0.90 and 0.95 which is above the cut-off of 0.90 (Schumacker & Lomax, 1996 ). Fourthly, the standardized root mean square residual (SRMR) value of 0.0392 which is situated well below the cut-off value of 0.05(Hu & Bentler, 1998 ), and the chi-square value of 3581.622 which is situated well below the cut-off value of 0.0005. Finally, the RMSEA (root mean square error of approximation) equals 0.04 with a 90% confidence interval between 0.03 and 0.05.

Analysis of mediation effect

The value and confidence interval are situated over but below the cut-off value of 0.1 which suggests not a great but a good fit. Factor analysis was verified by factor analysis using principal component analysis and only factors with an eigenvalue of 1 or more by orthogonal rotation method were selected. Factor loading was considered to be significant at 0.5 or more (Hair et al., 2006a , b ). As a result of the analysis, cumulative explanation for 72.4% of the total variance. Confirmatory factor analysis thus supported the differentiation of the three components Also we tested the confirmatory validity of the construct by testing whether the structural linkage of each square is greater than the mean variance extraction (AVE) of each structure. The AVE ranged from 0.52 to 0.53, reaching the recommended level of .50 for both Fornell and Larcker ( 1981 ). Therefore, all constructs showed sufficient convergent validity (see Table 3 ).

As shown in Table  4 , the AVE value of each variable has a higher value than that of other factors. Therefore, the discriminant validity of the proposed model can be judged as appropriate.

Means, standard deviations, and correlations among the study variables are shown in Table  5 .

The mean scores for the conceptual model were as follows for problem-solving ability (MD. 5.20, SD.1.08), innovative behavior (MD.5.20, SD.1.03), and opportunity recognition (MD. 5.14, SD. 1.06) conditions. The means of problem-solving ability, innovative behavior, and opportunity recognition were high. Furthermore, those variables correlated positively with each other.

Figure  1 showed that all paths and their significance levels are presented in Table  6 . The path between the latent variables problem-solving ability and innovative behavior was significant (p, 0.001), consistent with Hypotheses 1. In addition, there was innovative behavior and opportunity recognition (p, 0.01), this result provide empirical support for Hypothesis 2.

H3 proposed that Problem-solving ability is positively related to opportunity recognition. The results of the correlation analysis: The coefficient of problem solving and opportunity perception weakened from .717 to .444, but it is still partly mediated because it is still significant (C. R  = 7.604 ***). This supports H3 (see Table 6 ).

In order to verify the significance of the indirect effect, the bootstrapping must be performed in AMOS, and the actual significance test should be identified using two-tailed significance. As a result, the significance of indirect effect is 0.04 ( p  < 0.05), which is statistically significant (see Table  7 ).

Discussion and conclusion

We have tried to demonstrate the effects of behavior and its significance by differentiating from the general curriculum emphasizing cognitive effects as a model of problem solving ability emerging as innovative behavior through opportunity of university entrepreneurship education.. This supports the premise that entrepreneurship education can improve opportunities or processes through behavioral learning. The results of this study support the role of entrepreneurship education in creating opportunities for innovative behavior and problem solving abilities. Entrepreneurship education should provide different types of learning for new opportunities and focus on what is manifested in behavior.

In addition, based on previous research, we propose whether the following contents are well followed and whether it is effective. First, the emergence of innovative behavior in problem-solving abilities increases as the cognitive diversity of students with diverse majors and diverse backgrounds increases. Second, the more entrepreneurial learning experiences, the greater the chance of new opportunities. Third, it is necessary to investigate students’ problem solving style and problem-solving ability first, and then a teaching strategy based on this combination of systematic and effective theory and practice is needed. Of course, as demonstrated by many studies, it may be easier to enhance the effectiveness of opportunity recognition through cognitive learning. This is because it emphasizes the achievement of knowledge and understanding with acquiring skills and competence. This process, however, is not enough for entrepreneurship education. However, we do not support full team-based behavioral learning in the class designed by Michaelsen and Sweet ( 2008 ). As with the results of this study, problem solving ability is positively related to opportunity perception directly. As previously demonstrated in previous studies, problem solving ability can be enhanced by cognitive learning (Anderson et al., 2001 ; Charles & Lester, 1982 ).

Therefore, it has been demonstrated that it is more efficient to balance a certain level of cognitive learning and behavior learning in consideration of the level of students in a course. Also this study satisfies the need for empirical research by Lumpkin and Lichtenstein ( 2005 ) and Robinson et al. ( 2016 ) and others. This will help to improve understanding of how entrepreneurship training is linked to various learning models and their effectiveness and to design better courses for the future. Finally, this study sought to provide an awareness of entrepreneurship education as the best curriculum for solutions that evolved into innovative behaviors that create new values and ultimately represent new opportunities. This study shows that it can positively influence the social effect of creating new value, that is, not only the cognitive effect of general pedagogy, but also the innovation behavior. By providing this awareness, we have laid the groundwork for empirical research on entrepreneurship education in order to create more opportunities for prospective students in education through education and to expand their capabilities.

Limitation and future research

Indeed, the concepts presented here and the limitations of this study have important implications that can fruitfully be addressed in future research. First, we selected a sample of college students taking entrepreneurship training. However, since it is not the whole of Korean university students, it is difficult to extend the research results to all college students in Korea. Second, there is no precedent research on the role of innovation behavior as intermedia in college students. Therefore, we were forced to proceed as an exploratory study.

The ability to recognize opportunities can provide significant benefits that can remain firm and competitive in an ever-changing environment. Future research should therefore expand these insights and try to empirically test more ways in which entrepreneurship pedagogy teaches how learning methods can be integrated into venture creation and growth processes to help new process opportunities. By providing this study, we will help entrepreneurship education in the university to create more opportunities and expand the capacity of prospective members.

Alvarez, S. A., & Barney, J. B. (2005). How do entrepreneurs organize firms under conditions of uncertainty? Journal of Management, 31 (5), 776–793.

Article   Google Scholar  

Amabile, T. M. (1988). A model of creativity and innovation in organizations. Research in Organizational Behavior, 10 (1), 123–167.

Google Scholar  

Ancona, D. G., & Caldwell, D. F. (1992). Demography and design: Predictors of new product team performance. Organization Science, 3 (3), 321–341.

Anderson, P. M., & Anderson, P. M. (1995). Analysis of faulted power systems (Vol. 445). New York: IEEE press.

Anderson, L. W., Krathwohl, D. R., Airasian, P., Cruikshank, K., Mayer, R., Pintrich, P., & Wittrock, M. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy . New York: Longman Publishing.

Arbuckle, J. L., & Wothke, W. (2003). AMOS 5 user’s guide . Chicago: Smallwaters.

Ardichvili, A., Cardozo, R., & Ray, S. (2003a). A theory of entrepreneurial opportunity identification and development. Journal of Business Venturing, 18 (1), 105–123.

Ardichvili, A., Cardozo, R., & Ray, S. (2003b). A theory of entrepreneurial opportunity identification and development. Journal of Business Venturing, 18 (1), 105–123.

Baron, R. A., & Ensley, M. D. (2006). Opportunity recognition as the detection of meaningful patterns: Evidence from comparisons of novice and experienced entrepreneurs. Management Science, 52 (9), 1331–1344.

Barron, F., & Harrington, D. M. (1981). Creativity, intelligence, and personality. Annual review of psychology, 32 (1), 439–476.

Casson, M. (1982). The entrepreneur: An economic theory . Lanham: Rowman & Littlefield.

Charles, R., & Lester, F. (1982). Teaching problem solving: What, why & how . Palo Alto: Dale Seymour Publications.

Davidsson, P. (2015). Entrepreneurial opportunities and the entrepreneurship nexus: A re-conceptualization. Journal of Business Venturing, 30 (5), 674–695.

Dubickis, M., & Gaile-Sarkane, E. (2017). Transfer of know-how based on learning outcomes for development of open innovation. Journal of Open Innovation : Technology, market, and complexity , 3 (1), 4.

Dyer, J. H., Gregersen, H. B., & Christensen, C. (2008). Entrepreneur behaviors, opportunity recognition, and the origins of innovative ventures. Strategic Entrepreneurship Journal, 2 (4), 317–338.

D'zurilla, T. J., & Nezu, A. M. (1990). Development and preliminary evaluation of the social problem-solving inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 2 (2), 156.

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research , 39–50.

Gaglio, C. M., & Katz, J. A. (2001). The psychological basis of opportunity identification: Entrepreneurial alertness. Small Business Economics, 16 (2), 95–111.

Garavan, T. N., & O’Cinneide, B. (1994). Entrepreneurship education and training programmes: A review and evaluation-part 1. Journal of European Industrial Training, 18 (8), 3–12.

Gartner, W. B., & Carter, N. M. (2003). Entrepreneurial behavior and firm organizing processes. In Handbook of entrepreneurship research (pp. 195–221). New Mexico: Springer US.

Hair, E., Halle, T., Terry-Humen, E., Lavelle, B., & Calkins, J. (2006a). Children's school readiness in the ECLS-K: Predictions to academic, health, and social outcomes in first grade. Early Childhood Research Quarterly, 21 (4), 431–454.

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2006b). Multivariate Data Analysis (6th ed.). Upper Saddle River: Pearson Education, Inc..

Hayton, J. C., & Cholakova, M. (2012). The role of affect in the creation and intentional pursuit of entrepreneurial ideas. Entrepreneurship Theory and Practice, 36 (1), 41–68.

Hu, L. T., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3 (4), 424.

Inkinen, T. (2015). Reflections on the innovative city: Examining three innovative locations in a knowledge bases framework. Journal of Open Innovation : Technolodgy, market. Complexity, 1 (1), 8.

Isaksen, S. G. (1987). Frontiers of creativity research: Beyond the basics. Bearly Ltd.

Jabri, M. M. (1991). The development of conceptually independent subscales in the measurement of modes of problem solving. Educational and Psychological Measurement, 51 (4), 975–983.

Jonassen, D. H. (2004). Learning to solve problems: An instructional design guide (Vol. 6). Hoboken: John Wiley & Sons.

Kanter, R. M. (1983). The change masters: Binnovation and entrepreneturship in the American corporation. Touchstone Book.

Kanter, R. M. (1988). Three tiers for innovation research. Communication Research, 15 (5), 509–523.

Kim, H. C., Song, C. H., & An, B. R. (2016). A study on effects of personal characteristics on start-up opportunity and entrepreneurial intention of start-up. Korean Management Consulting review, 16 (3), 75–87.

Kim, S. A., Ryoo, H. Y., & Ahn, H. J. (2017). Student customized creative education model based on open innovation. Journal of Open Innovation : Technology, Market, and Complexity , 3 (1), 6.

Kim, T. H., & Roh, J. H. (2010). A Study of the Impact of Public Service Motivation on Innovative Behavior of Organizational Members. Korean Journal of Public Administration, 48(3).

Kirton, M. (1976). Adaptors and innovators: A description and measure. Journal of Applied Psychology, 61 (5), 622.

Kirzner, I. M. (1997). Entrepreneurial discovery and the competitive market process: An Austrian approach. Journal of Economic Literature, 35 (1), 60–85.

Ko, S., & Butler, J. E. (2003). Alertness, bisociative thinking ability, and discovery of entrepreneurial opportunities in Asian hi-tech firms.

Koestler, A. (1964). The act of creation: A study of the conscious and unconscious processes of humor, scientific discovery and art.

Lumpkin, G. T., & Dess, G. G. (2004). E-Business Strategies and Internet Business Models: How the Internet Adds Value. Organizational Dynamics, 33 (2), 161–173.

Lumpkin, G. T., & Lichtenstein, B. B. (2005). The role of organizational learning in the opportunity-recognition process. Entrepreneurship Theory and Practice, 29 (4), 451–472.

Manev, I. M., Gyoshev, B. S., & Manolova, T. S. (2005). The role of human and social capital and entrepreneurial orientation for small business performance in a transitional economy. International Journal of Entrepreneurship and Innovation Management, 5 (3–4), 298–318.

Michaelsen, L. K., & Sweet, M. (2008). The essential elements of team-based learning. New directions for teaching and learning, 2008 (116), 7–27.

Nunnally, J. C., & Bernstein, I. H. (1994). Validity. Psychometric theory, 99–132.

Paine, J. B., & Organ, D. W. (2000). The cultural matrix of organizational citizenship behavior: Some preliminary conceptual and empirical observations. Human Resource Management Review, 10 (1), 45–59.

Robinson, S., Neergaard, H., Tanggaard, L., & Krueger, N. F. (2016). New horizons in entrepreneurship education: from teacher-led to student-centered learning. Education+ Training, 58(7/8), 661–683.

Schumacker, R. E., & Lomax, R. G. (1996). A beginner's guide to structural equation modeling . Mahwah: Laurence Erlbaum Google Scholar.

Scott, S. G., & Bruce, R. A. (1994). Determinants of innovative behavior: A path model of individual innovation in the workplace. Academy of Management Journal, 37 (3), 580–607.

Shane, S. A. (2003). A general theory of entrepreneurship: The individual-opportunity nexus . Cheltenham: Edward Elgar Publishing.

Book   Google Scholar  

Shane, S., & Venkataraman, S. (2000). The promise of entrepreneurship as a field of research. Academy of Management Review, 25 (1), 217–226.

Siegel, S. M., & Kaemmerer, W. F. (1978). Measuring the perceived support for innovation in organizations. Journal of Applied Psychology, 63 (5), 553–562.

Spivack, G., Platt, J. J., & Shure, M. B. (1976). The problem-solving approach to adjustment . San Francisco: Jossey-Bass.

Stevenson, H., & Gumpert, D. (1985). The heart of entrepreneurship.

Stevenson, H. H. & J. C. Jarillo (1990). 'A paradigm of entrepreneurship: Entrepreneurial management', Strategic Management Journal, 11, pp. 17–27.

Ucbasaran, D., Westhead, P., & Wright, M. (2009). The extent and nature of opportunity identification by experienced entrepreneurs. Journal of Business Venturing, 24 (2), 99–115.

Van de Ven, A. H., & Angle, H. L. (1989). Suggestions for managing the innovation journey (No. 9). Strategic Management Research Center, University of Minnesota.

Venkataraman, S. (1997). The distinctive domain of entrepreneurship research. Advances in entrepreneurship, firm emergence and growth, 3 (1), 119–138.

Venkataraman, S., & Sarasvathy, S. D. (2001). Strategy and entrepreneurship: Outlines of an untold story.

Warner, M. (2002). Publics and counterpublics. Public Culture, 14 (1), 49–90.

Woodman, R. W., Sawyer, J. E., & Griffin, R. W. (1993). Toward a theory of organizational creativity. Academy of Management Review, 18 (2), 293–321.

Download references

Author information

Authors and affiliations.

Dept. of Technology Entrepreneurship (Graduate School), Dongguk University, 904 Chungmurogwn, Toegye-ro 36Gil, Jung-gu, Seoul, 100-272, South Korea

Ji Young Kim, Dae Soo Choi & Chang-Soo Sung

Yonsei School of Business, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul, 120-749, South Korea

Joo Y. Park

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Joo Y. Park .

Ethics declarations

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Kim, J.Y., Choi, D.S., Sung, CS. et al. The role of problem solving ability on innovative behavior and opportunity recognition in university students. J. open innov. 4 , 4 (2018). https://doi.org/10.1186/s40852-018-0085-4

Download citation

Received : 12 September 2017

Accepted : 22 January 2018

Published : 05 February 2018

DOI : https://doi.org/10.1186/s40852-018-0085-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Problem-solving ability
• Innovative behavior
• Opportunity recognition
• Entrepreneurship education

What Are Problem-Solving Skills? Definition and Examples

• Share on Twitter Share on Twitter
• Share on Facebook Share on Facebook
• Share on LinkedIn Share on LinkedIn

Forage puts students first. Our blog articles are written independently by our editorial team. They have not been paid for or sponsored by our partners. See our full  editorial guidelines .

Why do employers hire employees? To help them solve problems. Whether you’re a financial analyst deciding where to invest your firm’s money, or a marketer trying to figure out which channel to direct your efforts, companies hire people to help them find solutions. Problem-solving is an essential and marketable soft skill in the workplace. 

So, how can you improve your problem-solving and show employers you have this valuable skill? In this guide, we’ll cover:

Problem-Solving Skills Definition

Why are problem-solving skills important, problem-solving skills examples, how to include problem-solving skills in a job application, how to improve problem-solving skills, problem-solving: the bottom line.

Problem-solving skills are the ability to identify problems, brainstorm and analyze answers, and implement the best solutions. An employee with good problem-solving skills is both a self-starter and a collaborative teammate; they are proactive in understanding the root of a problem and work with others to consider a wide range of solutions before deciding how to move forward. 

Examples of using problem-solving skills in the workplace include:

• Researching patterns to understand why revenue decreased last quarter
• Experimenting with a new marketing channel to increase website sign-ups
• Brainstorming content types to share with potential customers
• Testing calls to action to see which ones drive the most product sales
• Implementing a new workflow to automate a team process and increase productivity

Problem-solving skills are the most sought-after soft skill of 2022. In fact, 86% of employers look for problem-solving skills on student resumes, according to the National Association of Colleges and Employers Job Outlook 2022 survey . 

It’s unsurprising why employers are looking for this skill: companies will always need people to help them find solutions to their problems. Someone proactive and successful at problem-solving is valuable to any team.

“Employers are looking for employees who can make decisions independently, especially with the prevalence of remote/hybrid work and the need to communicate asynchronously,” Eric Mochnacz, senior HR consultant at Red Clover, says. “Employers want to see individuals who can make well-informed decisions that mitigate risk, and they can do so without suffering from analysis paralysis.”

Showcase new skills

Build the confidence and practical skills that employers are looking for with Forage’s free job simulations.

Problem-solving includes three main parts: identifying the problem, analyzing possible solutions, and deciding on the best course of action.

>>MORE: Discover the right career for you based on your skills with a career aptitude test .

Research is the first step of problem-solving because it helps you understand the context of a problem. Researching a problem enables you to learn why the problem is happening. For example, is revenue down because of a new sales tactic? Or because of seasonality? Is there a problem with who the sales team is reaching out to? 

Research broadens your scope to all possible reasons why the problem could be happening. Then once you figure it out, it helps you narrow your scope to start solving it. 

Analysis is the next step of problem-solving. Now that you’ve identified the problem, analytical skills help you look at what potential solutions there might be.

“The goal of analysis isn’t to solve a problem, actually — it’s to better understand it because that’s where the real solution will be found,” Gretchen Skalka, owner of Career Insights Consulting, says. “Looking at a problem through the lens of impartiality is the only way to get a true understanding of it from all angles.”

Decision-Making

Once you’ve figured out where the problem is coming from and what solutions are, it’s time to decide on the best way to go forth. Decision-making skills help you determine what resources are available, what a feasible action plan entails, and what solution is likely to lead to success.

On a Resume

Employers looking for problem-solving skills might include the word “problem-solving” or other synonyms like “ critical thinking ” or “analytical skills” in the job description.

“I would add ‘buzzwords’ you can find from the job descriptions or LinkedIn endorsements section to filter into your resume to comply with the ATS,” Matthew Warzel, CPRW resume writer, advises. Warzel recommends including these skills on your resume but warns to “leave the soft skills as adjectives in the summary section. That is the only place soft skills should be mentioned.”

On the other hand, you can list hard skills separately in a skills section on your resume .

Forage Resume Writing Masterclass

Learn how to showcase your skills and craft an award-winning resume with this free masterclass from Forage.

Avg. Time: 5 to 6 hours

Skills you’ll build: Resume writing, professional brand, professional summary, narrative, transferable skills, industry keywords, illustrating your impact, standing out

In a Cover Letter or an Interview

Explaining your problem-solving skills in an interview can seem daunting. You’re required to expand on your process — how you identified a problem, analyzed potential solutions, and made a choice. As long as you can explain your approach, it’s okay if that solution didn’t come from a professional work experience.

“Young professionals shortchange themselves by thinking only paid-for solutions matter to employers,” Skalka says. “People at the genesis of their careers don’t have a wealth of professional experience to pull from, but they do have relevant experience to share.”

Aaron Case, career counselor and CPRW at Resume Genius, agrees and encourages early professionals to share this skill. “If you don’t have any relevant work experience yet, you can still highlight your problem-solving skills in your cover letter,” he says. “Just showcase examples of problems you solved while completing your degree, working at internships, or volunteering. You can even pull examples from completely unrelated part-time jobs, as long as you make it clear how your problem-solving ability transfers to your new line of work.”

Learn How to Identify Problems

Problem-solving doesn’t just require finding solutions to problems that are already there. It’s also about being proactive when something isn’t working as you hoped it would. Practice questioning and getting curious about processes and activities in your everyday life. What could you improve? What would you do if you had more resources for this process? If you had fewer? Challenge yourself to challenge the world around you.

Think Digitally

“Employers in the modern workplace value digital problem-solving skills, like being able to find a technology solution to a traditional issue,” Case says. “For example, when I first started working as a marketing writer, my department didn’t have the budget to hire a professional voice actor for marketing video voiceovers. But I found a perfect solution to the problem with an AI voiceover service that cost a fraction of the price of an actor.”

Being comfortable with new technology — even ones you haven’t used before — is a valuable skill in an increasingly hybrid and remote world. Don’t be afraid to research new and innovative technologies to help automate processes or find a more efficient technological solution.

Collaborate

Problem-solving isn’t done in a silo, and it shouldn’t be. Use your collaboration skills to gather multiple perspectives, help eliminate bias, and listen to alternative solutions. Ask others where they think the problem is coming from and what solutions would help them with your workflow. From there, try to compromise on a solution that can benefit everyone.

If we’ve learned anything from the past few years, it’s that the world of work is constantly changing — which means it’s crucial to know how to adapt . Be comfortable narrowing down a solution, then changing your direction when a colleague provides a new piece of information. Challenge yourself to get out of your comfort zone, whether with your personal routine or trying a new system at work.

Put Yourself in the Middle of Tough Moments

Just like adapting requires you to challenge your routine and tradition, good problem-solving requires you to put yourself in challenging situations — especially ones where you don’t have relevant experience or expertise to find a solution. Because you won’t know how to tackle the problem, you’ll learn new problem-solving skills and how to navigate new challenges. Ask your manager or a peer if you can help them work on a complicated problem, and be proactive about asking them questions along the way.

Career Aptitude Test

What careers are right for you based on your skills? Take this quiz to find out. It’s completely free — you’ll just need to sign up to get your results!

Step 1 of 3

Companies always need people to help them find solutions — especially proactive employees who have practical analytical skills and can collaborate to decide the best way to move forward. Whether or not you have experience solving problems in a professional workplace, illustrate your problem-solving skills by describing your research, analysis, and decision-making process — and make it clear that you’re the solution to the employer’s current problems. 

Image Credit: Christina Morillo / Pexels 

Related Posts

6 negotiation skills to level up your work life, how to build conflict resolution skills: case studies and examples, what is github uses and getting started, upskill with forage.

Build career skills recruiters are looking for.

• Research and Studies of Collaborative Problem Solving »

Research and Studies of Collaborative Problem Solving

This is the full list of published studies on Collaborative Problem Solving.

• Ashworth, K., Tapsak, S., & Li, S. T. (2012). Collaborative Problem Solving: Is empathy the active ingredient?   Graduate Student Journal of Psychology ,  14 , 83-92.
• Basso, R. V. J. & Graham, J. W. (2016). A longitudinal intervention study to reduce aggression by children ages 4-11 .  Journal of Behavior Therapy and Mental Health, 1(2) :12-23.
• Becker, K. D., Chorpita, B. F., & Daleiden, E. L. (2011). Improvement in symptoms versus functioning: How do our best treatments measure up?   Administration and Policy in Mental Health and Mental Health Services Research ,  38 (6), 440-458.
• Black, V., Bobier, C., Thomas, B., Prest, F., Ansley, C., Loomes, B., Eggleston, G., & Mountford, H. (2020). Reducing seclusion and restraint in a child and adolescent inpatient area: implementation of a collaborative problem-solving approach .  Australasian Psychiatry , 1-7.
• Bonnell, W., Alatishe, Y. A., & Hofner, A. (2014). The effects of a changing culture on a child and adolescent psychiatric inpatient unit .  Journal of the Canadian Academy of Child and Adolescent Psychiatry ,  23 (1), 65.
• Epstein, T., & Saltzman-Benaiah, J. (2010). Parenting children with disruptive behaviors: Evaluation of a Collaborative Problem Solving pilot program .  Journal of Clinical Psychology Practice ,  1 (1), 27-40.
• Ercole‐Fricke, E., Fritz, P., Hill, L. E., & Snelders, J. (2016). Effects of a Collaborative Problem‐Solving approach on an inpatient adolescent psychiatric unit .  Journal of Child and Adolescent Psychiatric Nursing ,  29 (3), 127-134.
• Gathright, M. M., Holmes, K. J., Morris, E. M., & Gatlin, D. A. (2016). An innovative, interdisciplinary model of care for inpatient child psychiatry: An overview .  The journal of behavioral health services & research ,  43 (4), 648-660.
• Greene, R. W., Ablon, J. S., & Goring, J. C. (2003). A transactional model of oppositional behavior: Underpinnings of the Collaborative Problem Solving approach .  Journal of Psychosomatic Research ,  55 (1), 67-75.
• Greene, R. W., Ablon, J. S., Goring, J. C., Raezer-Blakely, L., Markey, J., Monuteaux, M. C., ... & Rabbitt, S. (2004). Effectiveness of Collaborative Problem Solving in affectively dysregulated children with oppositional-defiant disorder: Initial findings .  Journal of consulting and clinical psychology ,  72 (6), 1157.
• Greene, R. W., Ablon, J. S., & Martin, A. (2006). Use of Collaborative Problem Solving to reduce seclusion and restraint in child and adolescent inpatient unit s.  Psychiatric Services ,  57 (5), 610-612.
• Hart, S. C., & DiPerna, J. C. (2017). Teacher beliefs and responses toward student misbehavior: Influence of cognitive skill deficit s.  Journal of applied school psychology ,  33 (1), 1-15.
• Heath, G. H., Fife‐Schaw, C., Wang, L., Eddy, C. J., Hone, M. J., & Pollastri, A. R. (2020). Collaborative Problem Solving reduces children's emotional and behavioral difficulties and parenting stress: Two key mechanisms . Journal of Clinical Psychology.
• Holmes, K. J., Stokes, L. D., & Gathright, M. M. (2014). The use of Collaborative Problem Solving to address challenging behavior among hospitalized children with complex trauma: A case series .  Residential Treatment for Children & Youth ,  31 (1), 41-62.
• Johnson, M., Östlund, S., Fransson, G., Landgren, M., Nasic, S., Kadesjö, B., ... & Fernell, E. (2012). Attention‐deficit/hyperactivity disorder with oppositional defiant disorder in Swedish children–an open study of Collaborative Problem Solving .  Acta Paediatrica ,  101 (6), 624-630.
• Kulkarni, G., Deshmukh, P., & Barzman, D. (2010). Collaborative Problem Solving (CPS) as a primary method of addressing acute pediatric pathological aggression along with other modalities .  Psychiatric quarterly ,  81 (2), 167-175.
• Martin, A., Krieg, H., Esposito, F., Stubbe, D., & Cardona, L. (2008). Reduction of restraint and seclusion through Collaborative Problem Solving: A five-year prospective inpatient study .  Psychiatric Services ,  59 (12), 1406-1412.
• Mohr, W. K., Martin, A., Olson, J. N., Pumariega, A. J., & Branca, N. (2009). Beyond point and level systems: Moving toward child‐centered programming .  American Journal of Orthopsychiatry ,  79 (1), 8-18.
• Open Arms Program of the Cambridge Hospital Child Assessment Unit, Cambridge, Massachusetts. (2003). 2003 APA Gold Award: A more compassionate model for treating children with severe mental disturbances .  Psychiatric Services ,  54 (11), 1529-1531.
• Pollastri, A. R., Epstein, L. D., Heath, G. H., & Ablon, J. S. (2013). The Collaborative Problem Solving approach: Outcomes across settings .  Harvard Review of Psychiatry ,  21 (4), 188-199.
• Pollastri, A. R., Lieberman, R. E., Boldt, S. L., & Ablon, J. S. (2016). Minimizing seclusion and restraint in youth residential and day treatment through site-wide implementation of Collaborative Problem Solving .  Residential Treatment for Children & Youth ,  33 (3-4), 186-205.
• Pollastri, A. R., Rosenbaum, C., & Ablon, J. S. (2019). Disruptive Behavior Disorders . In  The Massachusetts General Hospital Guide to Learning Disabilities  (pp. 207-220). Humana Press, Cham.
• Pollastri, A.R., Wang, L., Eddy C.J., Ablon, J. S. An open trial of Collaborative Problem Solving in a naturalistic outpatient setting . Clinical Child Psychology and Psychiatry 2022; 28(2): 512-524.
• Pollastri, A. R., Wang, L., Raftery-Helmer, J. N., Hurley, S., Eddy, C. J., Sisson, J., Thompson, N., & Ablon, J. S. (2022). Development and evaluation of an audio coding system for assessing providers’ integrity to Collaborative Problem Solving in youth-service settings .  Professional Psychology: Research and Practice , online first.
• Pollastri, A. R., Wang, L., Youn, S. J., Ablon, J. S., & Marques, L. (2020). The value of implementation frameworks: Using the active implementation frameworks to guide system‐wide implementation of Collaborative Problem Solving . Journal of Community Psychology .
• Regan, K. M., Curtin, C., & Vorderer, L. (2006). Paradigm shifts in inpatient psychiatric care of children: approaching child‐and family‐centered care .  Journal of Child and Adolescent Psychiatric Nursing ,  19 (1), 29-40.
• Schaubman, A., Stetson, E., & Plog, A. (2011). Reducing teacher stress by implementing Collaborative Problem Solving in a school setting .  School Social Work Journal ,  35 (2), 72-93.
• Stetson, E. A., & Plog, A. E. (2016).  Collaborative Problem Solving in schools: Results of a year-long consultation project .  School Social Work Journal ,  40 (2), 17-36.
• Stewart, S. L., Rick, J., Currie, M., & Rielly, N. (2009).  Collaborative Problem Solving approach in clinically referred children: A residential program evaluation .  Unpublished manuscript, Department of Applied Research and Education Child and Parent Resource Institute, London, Ontario, Canada .
• Stoll, S. J., Hartman, J. D., Paxton, D., Wang, L., Ablon, J. S., Perry, B. D., & Pollastri, A. R. (2023). De-Implementing a Point and Level System in Youth Residential Care without Increased Safety Risk: A Case Study . Residential Treatment for Children & Youth. Online first.
• Valenkamp, M., Delaney, K., & Verheij, F. (2014). Reducing seclusion and restraint during child and adolescent inpatient treatment: Still an underdeveloped area of research .  Journal of Child and Adolescent Psychiatric Nursing ,  27 (4), 169-174.
• Wang, L., & Pollastri, A. R. (2019).   User’s Guide to the Collaborative Problem Solving Adherence & Impact Measures (CPS-AIMs) . Think:Kids, Massachusetts General Hospital, Harvard Medical School.
• Wang, L., Pollastri, A. R., Vuijk, P. J., Hill, E. N., Lee, B. A., Samkavitz, A., … & Doyle, A. E. (2019). Reliability and Validity of the Thinking Skills Inventory, a Screening Tool for Cross-Diagnostic Skill Deficits Underlying Youth Behavioral Challenges .  Journal of Psychopathology and Behavioral Assessment ,  41 (1), 144-159.
• Wang, L., Stoll, S. J., Eddy, C. J., Hurley, S., Sisson, J., Thompson, N., Raftery-Helmer, J. N., Ablon, J. S., Pollastri, A. R. (2023). Pragmatic fidelity measurement in youth service settings .  Implementation Research and Practice ,  4,  1-13.
• Wang, L., Stoll, S., Hone, M., Ablon, J. S., & Pollastri, A. R. (2022). Effects of a Collaborative Problem Solving parent group on parent and child outcomes . Child & Family Behavior Therapy, 44(4), 241-258.

Request a Research Paper

" * " indicates required fields

Privacy Overview

• DOI: 10.52902/kjsc.2024.28.121
• Corpus ID: 269039411

Correlation between Communication Competence, Problem-Solving Skills, Clinical Competence, and Critical Thinking Competence on Person-Centered Care Competence of Nursing Students in who Experienced Clinical Practice

• Mi Young Moon
• Published in Forum of Public Safety and… 30 March 2024
• Education, Medicine

Related Papers

Showing 1 through 3 of 0 Related Papers

Problem-solving skills and how to improve them (with examples)

What’s life without its challenges? All of us will at some point encounter professional and personal hurdles. That might mean resolving a conflict with coworkers or making a big life decision. With effective problem solving skills, you’ll find tricky situations easier to navigate, and welcome challenges as opportunities to learn, grow and thrive. 

In this guide, we dive into the importance of problem solving skills and look at examples that show how relevant they are to different areas of your life. We cover how to find creative solutions and implement them, as well as ways to refine your skills in communication and critical thinking. Ready to start solving problems? Read on.

What is problem solving? 

Before we cover strategies for improving problem solving skills, it’s important to first have a clear understanding of the problem solving process. Here are the steps in solving a problem:

• Recognise the issue you are facing 
• Take a look at all the information to gain insights
• Come up with solutions
• Look at the pros and cons of each solution and how it might play out
• Plan, organise and implement your solution
• Continuously assess the effectiveness of the solution and make adjustments as needed

Problem solving skills

There’s more to problem solving than coming up with a quick fix. Effective problem solving requires wide range of skills and abilities, such as:

• Critical thinking: the ability to think logically, analyse information and look at situations from different perspectives.
• Creativity: being able to come up with innovative, out-of-the-box solutions.
• Decision-making:  making informed choices by considering all the available information.
• Communication:  being able to express ideas clearly and effectively.
• Analytical skills: breaking down complex problems into smaller parts and examining each one.
• Time management:  allocating time and resources effectively to address problems.
• Adaptability: being open to change and willing to adjust strategies.
• Conflict resolution:  skillfully managing conflicts and finding solutions that work for all.

Examples of problem solving skills

Problem solving skills in the workplace are invaluable, whether you need them for managing a team, dealing with clients or juggling deadlines. To get a better understanding of how you might use these skills in real-life scenarios, here are some problem solving examples that are common in the workplace.

• Analytical thinking

Analytical thinking is something that comes naturally to some, while others have to work a little harder. It involves being able to look at problem solving from a logical perspective, breaking down the issues into manageable parts. 

Example scenarios of analytical thinking

Quality control: in a manufacturing facility, analytical thinking helps identify the causes of product defects in order to pinpoint solutions.

Market research: marketing teams rely on analytical thinking to examine consumer data, identify market trends and make informed decisions on ad campaigns.

• Critical thinking

Critical thinkers are able to approach problems objectively, looking at different viewpoints without rushing to a decision. Critical thinking is an important aspect of problem solving, helping to uncover biases and assumptions and weigh up the quality of the information before making any decisions. 

Example scenarios of critical thinking

• Strategic planning:  in the boardroom, critical thinking is important for assessing economic trends, competitor threats and more. It guides leaders in making informed decisions about long-term company goals and growth strategies.
• Conflict resolution: HR professionals often use critical thinking when dealing with workplace conflicts. They objectively analyse the issues at hand and find an appropriate solution.

Decision-making

Making decisions is often the hardest part of problem solving. How do you know which solution is the right one? It involves evaluating information, considering potential outcomes and choosing the most suitable option. Effective problem solving relies on making well-informed decisions.

Example scenarios of decision-making

• Budget allocation: financial managers must decide how to allocate resources to various projects or departments. 
• Negotiation:  salespeople and procurement professionals negotiate terms, pricing and agreements with clients, suppliers and partners.

Research skills

Research skills are pivotal when it comes to problem solving, to ensure you have all the information you need to make an informed decision. These skills involve searching for relevant data, critically evaluating information sources, and drawing meaningful conclusions. 

Example scenarios of research skills

• Product development: a tech startup uses research skills to conduct market research to identify gaps and opportunities in the market. 
• Employee engagement:  an HR manager uses research skills to conduct employee surveys and focus groups.

A little creative flair goes a long way. By thinking outside the box, you can approach problems from different angles. Creative thinking involves combining existing knowledge, experiences and perspectives in new and innovative ways to come up with inventive solutions. 

Example scenarios of creativity

• Cost reduction: creative problem solvers within a manufacturing company might look at new ways to reduce production costs by using waste materials.
• Customer experience: a retail chain might look at implementing interactive displays and engaging store layouts to increase customer satisfaction and sales.

Collaboration

It’s not always easy to work with other people, but collaboration is a key element in problem solving, allowing you to make use of different perspectives and areas of expertise to find solutions.

Example scenarios

• Healthcare diagnosis: in a hospital setting, medical professionals collaborate to diagnose complex medical cases.
• Project management: project managers coordinate efforts, allocate resources and address issues that may arise during a project's lifecycle.

Conflict Resolution

Being able to mediate conflicts is a great skill to have. It involves facilitating open communication, understanding different perspectives and finding solutions that work for everyone. Conflict resolution is essential for managing any differences in opinion that arise.

Example scenarios of conflict resolution

• Client dispute: a customer might be dissatisfied with a product or service and demand a refund. The customer service representative addresses the issue through active listening and negotiation to reach a solution.
• Project delay: a project manager might face resistance from team members about a change in project scope and will need to find a middle ground before the project can continue.

Risk management

Risk management is essential across many workplaces. It involves analysing potential threats and opportunities, evaluating their impact and implementing strategies to minimise negative consequences. Risk management is closely tied to problem solving, as it addresses potential obstacles and challenges that may arise during the problem solving process.

Example scenarios of risk management

• Project risk management: in a construction project, risk management involves identifying potential delays, cost overruns and safety hazards. Risk mitigation strategies are developed, such as scheduling buffers and establishing safety protocols. 
• Financial risk management: in financial institutions, risk management assesses and manages risks associated with investments and lending.

Communication

Effective communication is a skill that will get you far in all areas of life. When it comes to problem solving, communication plays an important role in facilitating collaboration, sharing insights and ensuring that all stakeholders have the same expectations. 

Example scenarios of communication

• Customer service improvement:  in a retail environment, open communication channels result in higher customer satisfaction scores.
• Safety enhancement:  in a manufacturing facility, a robust communication strategy that includes safety briefings, incident reporting and employee training helps minimise accidents and injuries.

How to improve problem solving skills 

Ready to improve your problem solving skills? In this section we explore strategies and techniques that will give you a head start in developing better problem solving skills. 

Adopt the problem solving mindset

Developing a problem solving mindset will help you tackle challenges effectively . Start by accepting problems as opportunities for growth and learning, rather than as obstacles or setbacks. This will allow you to approach every challenge with a can-do attitude.

Patience is also essential, because it will allow you to work through the problem and its various solutions mindfully. Persistence is also important, so you can keep adapting your approach until you find the right solution.

Finally, don’t forget to ask questions. What do you need to know? What assumptions are you making? What can you learn from previous attempts? Approach problem solving as an opportunity to  acquire new skills . Stay curious, seek out solutions, explore new possibilities and remain open to different problem solving approaches.

Understand the problem

There’s no point trying to solve a problem you don’t understand. To analyse a problem effectively, you need to be able to define it. This allows you to break it down into smaller parts, making it easier to find causes and potential solutions. Start with a well-defined problem statement that is precise and specific. This will help you focus your efforts on the core issue, so you don’t waste time and resources on the wrong concerns.

Strategies for problem analysis

• Start with the problem statement and ask ‘Why?’ multiple times to dig deeper.
• Gather relevant data and information related to the problem. 
• Include those affected by the problem in the analysis process.
• Compare the current problem with similar situations or cases to gain valuable insights.
• Use simulations to explore potential outcomes of different solutions.
• Continuously gather feedback during the problem solving process. 

Develop critical thinking and creativity skills

Critical thinking and creativity are both important when it comes to looking at the problem objectively and thinking outside the box. Critical thinking encourages you to question assumptions, recognise biases and seek evidence to support your conclusions. Creative thinking allows you to look at the problem from different angles to reveal new insights and opportunities.

Enhance research and decision-making skills

Research and decision-making skills are pivotal in problem solving as they enable you to gather relevant information, analyse options and choose the best course of action. Research provides the information and data needed, and ensures that you have a comprehensive understanding of the problem and its context. Effective decision-making is about selecting the solution that best addresses the problem.

Strategies to improve research and decision-making skills

• Clearly define what you want to achieve through research.
• Use a variety of sources, including books, articles, research papers, interviews, surveys and online databases.
• Evaluate the credibility and reliability of your information sources.
• Incorporate risk assessment into your decision-making process. 
• Seek input from experts, colleagues and mentors when making important decisions. 
• After making decisions, reflect on the outcomes and lessons learned. Use this to improve your decision-making skills over time.

Strengthen collaboration skills

Being able to work with others is one of the most important skills to have at work. Collaboration skills enable everyone to work effectively as a team, share their perspectives and collectively find solutions. 

Tips for improving teamwork and collaboration

• Define people’s roles and responsibilities within the team. 
• Encourage an environment of open communication where team members feel comfortable sharing ideas.
• Practise active listening by giving full attention to others when they speak. 
• Hold regular check-in sessions to monitor progress, discuss challenges and make adjustments as needed.
• Use collaboration tools and platforms to facilitate communication and document progress. 
• Acknowledge and celebrate team achievements and milestones. 

Learn from past experiences

Once you’ve overcome a challenge, take the time to look back with a critical eye. How effective was the outcome? Could you have tweaked anything in your process? Learning from past experiences is important when it comes to problem solving. It involves reflecting on both successes and failures to gain insights, refine strategies and make more informed decisions in the future. 

Strategies for learning from past mistakes

• After completing a problem solving effort, gather your team for a debriefing session. Discuss what went well and what could have been better.
• Conduct a SWOT analysis (Strengths, Weaknesses, Opportunities, Threats) of resolved problems. 
• Evaluate the outcomes of past solutions. Did they achieve the desired results? 
• Commit to continuous learning and improvement. 

Leverage problem solving tools and resources

Problem-solving tools and resources are a great help when it comes to navigating complex challenges. These tools offer structured approaches, methodologies and resources that can streamline the process. 

Tools and resources for problem solving

• Mind mapping:  mind maps visually organise ideas, concepts and their relationships. 
• SWOT (Strengths, Weaknesses, Opportunities, Threats) Analysis:  helps in strategic planning and decision-making.
• Fishbone diagram (Ishikawa Diagram): this tool visually represents the potential root causes of a problem, helping you identify underlying factors contributing to an issue.
• Decision matrices:  these assist in evaluating options by assigning weights and scores to criteria and alternatives.
• Process flowcharts:  these allow you to see the steps of a process in sequence, helping identify where the problem is occuring.
• Decision support software:  software applications and tools, such as data analytics platforms, can help in data-driven decision-making and problem solving.
• Online courses and training: allow you to acquire new skills and knowledge.

Regular practice

Practice makes perfect! Using your skills in real life allows you to refine them, adapt to new challenges and build confidence in your problem solving capabilities. Make sure to try out these skills whenever you can.

Practical problem solving exercises 

• Do puzzles, riddles and brainteasers regularly. 
• Identify real-life challenges or dilemmas you encounter and practice applying problem solving techniques to these situations.
• Analyse case studies or scenarios relevant to your field or industry. 
• Regularly review past problem solving experiences and consider what you learned from them. 
• Attend workshops, webinars or training sessions focused on problem solving. 

How to highlight problem solving skills on a resumé

Effectively showcasing your problem solving skills on your resumé is a great way to demonstrate your ability to address challenges and add value to a workplace. We'll explore how to demonstrate problem solving skills on your resumé, so you stand out from the crowd.

Incorporating problem solving skills in the resumé summary

A resumé summary is your introduction to potential employers and provides an opportunity to succinctly showcase your skills. The resumé summary is often the first section employers read. It offers a snapshot of your qualifications and sets the tone for the rest of your resumé.

Your resumé summary should be customised for different job applications, ensuring that you highlight the specific problem solving skills relevant to the position you’re applying for.

Example 1: Project manager with a proven track record of solving complex operational challenges. Skilled in identifying root causes, developing innovative solutions and leading teams to successful project completion.

Example 2:  Detail-oriented data analyst with strong problem solving skills. Proficient in data-driven decision-making, quantitative analysis and using statistical tools to solve business problems.

Highlighting problem solving skills in the experience section

The experience section of your resumé presents the perfect opportunity to demonstrate your problem solving skills in action. 

• Start with action verbs: begin each bullet point in your job descriptions with strong action verbs such as, analysed, implemented, resolved and optimised.
• Quantify achievements: use numbers and percentages to illustrate the impact of your solutions. For example: Increased efficiency by 25% by implementing a new workflow process.
• Emphasise challenges: describe the specific challenges or problems you faced in your roles. 
• Solution-oriented language: mention the steps you took to find solutions and the outcomes achieved.

Including problem solving skills in the skills section

The skills section of your resumé should showcase your top abilities, including problem solving skills. Here are some tips for including these skills.

• Use a subsection:  within your skills section, you could create a subsection specifically dedicated to problem solving skills – especially if the role calls for these skills.
• Be specific: when listing problem solving skills, be specific about the types of role-related problems you can address. 
• Prioritise relevant skills:  tailor the list of problem solving skills to match the requirements of the job you're applying for. 

Examples of problem solving skills to include:

• Creative problem solving
• Decision making
• Root cause analysis
• Strategic problem solving
• Data-driven problem solving
• Interpersonal conflict resolution
• Adaptability
• Communication skills
• Problem solving tools
• Negotiation skills

Demonstrating problem solving skills in project sections or case studies

Including a dedicated section for projects or case studies in your resumé allows you to provide specific examples of your problem solving skills in action. It goes beyond simply listing skills, to demonstrate how you are able to apply those skills to real-world challenges.

Example – Data Analysis

Case Study: Market Expansion Strategy

• Challenge:  the company was looking to expand into new markets but lacked data on consumer preferences and market dynamics.
• Solution: conducted comprehensive market research, including surveys and competitor analysis. Applied this research to identify target customer segments and developed a data-driven market-entry strategy.
• Result:  successfully launched in two new markets, reaching our target of 30% market share within the first year.

Using problem solving skills in cover letters

A well-crafted cover letter is your first impression on any potential employer. Integrating problem solving skills can support your job application by showcasing your ability to address challenges and contribute effectively to their team. Here’s a quick run-down on what to include:

• Begin your cover letter by briefly mentioning the position you're applying for and your enthusiasm for it.
• Identify a specific challenge or issue that the company may be facing, to demonstrate your research and understanding of their needs.
• Include a brief story or scenario from your past experiences where you successfully applied problem solving skills to address a similar challenge. 
• Highlight the positive outcomes or results achieved through your problem solving efforts. 
• Explain how your skills make you the ideal person to address their specific challenges.

Problem solving skills are essential in all areas of life, enabling you to overcome challenges, make informed decisions, settle conflicts and drive innovation. We've explored the significance of problem solving skills and how to improve, demonstrate and leverage them effectively. It’s an ever-evolving skill set that can be refined over time. 

By actively incorporating problem solving skills into your day-to-day, you can become a more effective problem solver at work and in your personal life as well.

What are some common problem solving techniques?

Common problem solving techniques include brainstorming, root cause analysis, SWOT analysis, decision matrices, the scientific method and the PDCA (Plan-Do-Check-Act) cycle. These techniques offer structured approaches to identify, analyse and address problems effectively.

How can I improve my critical thinking skills?

Improving critical thinking involves practising skills such as analysis, evaluation and problem solving. It helps to engage in activities like reading, solving puzzles, debating and self-reflection.

What are some common obstacles to problem solving?

Common obstacles to problem solving include biases, lack of information or resources, and resistance to change. Recognising and addressing these obstacles is essential for effective problem solving.

How can I overcome resistance to change when implementing a solution?

To overcome resistance to change, it's essential to communicate the benefits of the proposed solution clearly, involve stakeholders in the decision-making process, address concerns and monitor the implementation's progress to demonstrate its effectiveness.

How can problem solving skills benefit my career?

Problem solving skills are highly valuable in a career as they enable you to navigate challenges, make informed decisions, adapt to change and contribute to innovation and efficiency. These skills enhance your professional effectiveness and can lead to career advancement and increased job satisfaction.

Top search terms

Explore related topics, subscribe to career advice.

Can AI Match Human Ingenuity in Creative Problem-Solving?

When ChatGPT and other large language models began entering the mainstream two years ago, it quickly became apparent the technology could excel at certain business functions, yet it was less clear how well artificial intelligence could handle more creative tasks.

Sure, generative AI can summarize the content of an article, identify patterns in data, and produce derivative work—say, a song in the style of Taylor Swift or a poem in the mood of Langston Hughes—but can the technology develop truly innovative ideas?

Specifically, Harvard Business School Assistant Professor Jacqueline Ng Lane was determined to find out “how AI handled open-ended problems that haven’t been solved yet—the kind where you need diverse expertise and perspectives to make progress.”

In a working paper published in the journal Organization Science , Lane and colleagues compare ChatGPT’s creative potential to crowdsourced innovations produced by people. Ultimately, the researchers found that both humans and AI have their strengths—people contribute more novel suggestions while AI creates more practical solutions—yet some of the most promising ideas are the ones people and machines develop together.

Lane cowrote the paper with Léonard Bouissioux, assistant professor at the University of Washington’s Foster School of Business; Miaomiao Zhang, an HBS doctoral student, Karim Lakhani, the Dorothy & Michael Hintze Professor of Business Administration at HBS; and Vladimir Jacimovic, CEO and founder of ContinuumLab.ai and executive fellow at HBS.

Crowdsourcing people for ‘moonshots’

Any innovation process usually starts with brainstorming, says Lane, whose research has long looked at how creative ideas are produced.

“You start with defining the problem, then you generate ideas, then you evaluate them and choose which ones to implement.”

“It’s like a funnel,” she says. “You start with defining the problem, then you generate ideas, then you evaluate them and choose which ones to implement.”

Research has shown that crowdsourcing can be an effective way to generate initial ideas. However, the approach can be time-consuming and expensive. Creative teams typically offer incentives to respondents for their ideas. Then teams often must wait for input and then comb through ideas to come up with the most promising leads.

An off-the-shelf large language model such as ChatGPT, however, is free or low cost for end users, and can generate an infinite number of ideas quickly, Lane says. But are the ideas any good?

To find out, Lane and her fellow researchers asked people to come up with business ideas for the sustainable circular economy, in which products are reused or recycled to make new products. They disseminated a request on an online platform, offering $10 for participating and $1,000 for the best idea. Here’s part of their request:

We would like you to submit your circular economy idea, which can be a unique new idea or an existent idea that is used in the industry.

Here is an example: Car sharing in order to reduce the carbon footprint associated with driving. …

Submit your real-life use cases on how companies can implement the circular economy in their businesses. New ideas are also welcome, even if they are “moonshots.”

Seeking creative ideas from ChatGPT

The researchers asked for ideas that would involve “sharing, leasing, reusing, repairing, refurbishing [or] recycling existing materials and products as long as possible.” Suggestions would be scored for uniqueness, environmental benefits, profit potential, and feasibility.

Some 125 people replied with contributions, offering insights from a variety of industries and professional backgrounds. One, for example, proposed a dynamic pricing algorithm for supermarkets to cut down on food waste, while another suggested a mobile app that could store receipts to reduce paper waste.

At the same time, the research team employed prompt engineering techniques to craft a variety of AI prompts. Using these carefully designed prompts, they generated several hundred additional solutions through ChatGPT. The team strategically modified their prompts to:

• Challenge the model to create more ideas.
• Mimic the perspective of someone from a particular industry, job title, and place—a persona.
• Remind the model to provide ideas that reflect the scoring criteria.

The team then recruited some 300 evaluators well-versed in the circular economy to evaluate a randomized selection of the ideas based on the scoring criteria.

People are creative, but AI ideas are more feasible

The evaluators judged the human solutions as more novel, employing more unique “out of the box” thinking. However, they found the AI-generated ideas to be more valuable and feasible.

For example, one participant from Africa proposed creating interlocking bricks using foundry dust and waste plastic, creating a new construction material and cutting down on air pollution at the same time. “The evaluators said, ‘Wow, this is really innovative, but it would never work,’” Lane says.

“We were surprised at how powerful these technologies were.”

One ChatGPT response, meanwhile, created an idea to convert food waste into biogas, a renewable energy source that could be used for electricity and fertilizer. Not the most novel idea, the researchers noted, but one that could be implemented and might show a clear financial return.

“We were surprised at how powerful these technologies were,” Lane says, “especially in these early stages in the creative process.”

How to reach the best solutions

The “best” ideas, Lane says, may come from those in which humans and AI collaborate, with people engineering prompts and continually working with AI to develop more original ideas.

“We consistently achieved higher quality results when AI would come up with an idea and then we had an instruction that said: Make sure before you create your next idea, it’s different from all the ones before it,” Lane explains.

Additional prompts increased the novelty of the ideas, generating everything from waste-eating African flies to beverage containers tracked by smart chips that instantly pay consumers for recycling them.

Based on the findings, the researchers suggest business leaders keep a few points in mind when implementing AI to develop creative solutions:

• Knowing how to ask the right questions is important. Organizations might want to invest in cultivating an “AI-literate” workforce that can understand the capabilities and limitations of AI to generate the most successful ideas.
• Organizations should resist the temptation to rely excessively on AI. That could “dumb down” the overall level of creative output over time, leading to more incremental improvements than radical breakthroughs, the team says.
• People should view generative AI models as collaborative tools. In a sequential approach, humans could brainstorm solutions, then submit them to AI to refine them and increase their value and feasibility. Alternatively, humans could work more iteratively with AI, constantly shaping and improving the ideas it provides.

The most productive way to use generative AI, the research suggests, is to combine the novelty that people excel at with the practicality of the machine. Says Lane, “We still need to put our minds toward being forward-looking and envisioning new things as we are guiding the outputs of AI to create the best solutions.”

Rapport: The Hidden Advantage That Women Managers Bring to Teams

• 11 Jun 2024
• In Practice

The Harvard Business School Faculty Summer Reader 2024

How transparency sped innovation in a $13 billion wireless sector.

• 24 Jan 2024

Why Boeing’s Problems with the 737 MAX Began More Than 25 Years Ago

• 27 Jun 2016

These Management Practices, Like Certain Technologies, Boost Company Performance

• Problems and Challenges
• Innovation and Invention
• Disruptive Innovation
• Search Technology
• Information Technology
• Information

Sign up for our weekly newsletter