1The reality of the language learning environment is most of the time very different from the ideal situations that are presented in the prototypes enthusiastic researchers show us. Students in French universities, and especially students in languages and the human sciences, are more often than not reluctant to accept changes associated with the implementation of a new learning system, such as blended learning. The best learning project, even if it is theoretically valid and combines the latest technological innovations, can do nothing against the beliefs and representations of human beings. This means that, when implementing a learning system, it is necessary to evaluate the impact on learners (Narcy-Combes M.-F. 2008a). It also pertains to the epistemological responsibility of the teacher/researcher to ensure the validity of the implemented system (Narcy-Combes J.-P. 2005).
2The aim of this study is to determine the validity of the blended language learning environment implemented in Nantes University by measuring its effects on second language acquisition. The evaluation is carried out at three levels: evaluation of student language levels before and after the programme, evaluation of student involvement pertaining to their use of the on-line platform and finally, analysis of student discourse and interactions in class to determine if language learning occurs. The article will begin by examining the context of the blended learning project including the rationale for implementing such a programme and a description of the project. It will then outline the theoretical background for second language acquisition which forms the foundation of the blended learning evaluation project. Having set out the theoretical view, the article will then focus on the results of the evaluation and highlight future developments.
3The blended language learning programme, in which this research was conducted, concerns 660 students enrolled in their first year of an undergraduate Applied Languages course at Nantes University. Students can study up to three languages including English, which is compulsory, and Spanish, German, Italian, Portuguese, Russian, Arabic, Chinese or Japanese. They also study business-related subjects including law, economics and management. The audience is primarily female (66%) with an educational background in economics (32%) and languages (27%). Their social background is mixed: 25% middle-class, 22% lower-middle class and 15% working class. However, the socioeconomic background is not a significant variable. This student profile has remained consistent throughout the last decade, as shown by a previous study (Narcy-Combes 2003) carried out in 2001.
4Despite the programme attracting increasing student numbers in the first year (up by 24% between 2006 and 2008), it is also characterised by a 60% drop-out rate at the end of the first year. This has raised questions for the team of teachers as to why students fail and what can be done to reduce the problem. The size of language classes, which can vary from 45 to 60 students per class, is a contributing factor to student failure.
5The introduction of a blended learning programme in January 2009 was a response to this problem. In this case, combining a task-based approach with distance learning through a learning platform (Moodle) and tutorials was worked out as a specific solution to cope with large groups of – often – demotivated students who choose this course as a last resort. The hypotheses are that first providing real-life tasks, which are directly linked to the students’ career prospects, will trigger their motivation and involvement and that secondly face-to-face tutorials in smaller groups will favour peer and student to teacher interactions which will promote second language acquisition.
6The blended language learning programme implemented at Nantes University combines two hours of face-to-face instruction with 2-3 hours computer- mediated instruction. The computer-mediated instruction is self-paced, so the actual duration depends on the rhythm of the student. The face-to-face instruction is broken down into two distinct classes: students meet with the teacher for one hour as a whole class of 45 students and one hour in tutorials of fifteen students. The computer-mediated instruction is based on Moodle, an open source virtual learning platform, and comprises two distinct spaces:
- a space containing lesson materials, i.e., documents with detailed instructions relating to the preparatory work to be carried out and task objectives;
- a self-training centre which contains interactive grammar, vocabulary and listening exercises.
7Students have automatic access to the lesson space, but must register to have access to the self-training centre. Students worked on eight tasks during a twelve week semester. Tasks were in the form of real-life business scenarios (Roots-Buck 2005) and involved either an oral or a written production as highlighted in the example below.
Example Task Brief
The company that you work for wants to look at the concept of outsourcing as a means of reducing manufacturing or service costs. Given your excellent knowledge of English, you’ve been asked to do the research about this subject and to write a formal written report on the pros and cons of the concept for your company.
Before you start, you need to invent your company. There has to be a good reason for you to work in English, so your company will probably be a multinational.
8Learners go through four phases: (i) on-line self-paced learning to acquire background information on the task (for example, by doing preparatory work activities such as reading and listening comprehension and vocabulary exercises); (ii) collaborative learning and problem solving in groups of 3-4 students to prepare the task outcome; (iii) face-to-face learning focused on active learning and application of tasks or meta-reflection based on personalised feedback and support from teachers; (iv) on-line targeted language learning activities in the self-training centre (students work on those language areas they have most difficulty with as highlighted by the teacher’s feedback).
9The advantages for learners are numerous: they are able to work at their own pace and have more opportunities to practice, both orally and in writing and benefit from more personalised feedback and advice from their teacher, both during the tutorials and on-line.
10We take a constructivist stance to language learning and evaluation (Bruner 2000). Accordingly, learning is not transmitted from teacher to learner, but rather constructed by the learner through social interaction and individual involvement and reflection (Lantolf 2000, Little 2007). In this sense, the learner is expected to take charge of his or her own learning and knowledge is viewed as a set of ’working hypotheses’ (Little 2007) that are always subject to refinement, change, rejection and replacement (Little 2007).
11All acquisition theories agree on the essential role of communicative language use for language acquisition: input or content is re-embedded in the task where the learner will construct his knowledge through linguistically-mediated interactions. In this approach, language acquisition is realised through a collaborative process whereby learners appropriate the language of the interaction as their own, for their own purposes and in turn build grammatical, expressive and cultural competence (Lantolf 2000). The mental effort he/she needs to produce language stimulates deep language processing (Swain in Lantolf 2000). Swain (2000) proposes three different functions of output in second language acquisition: it promotes ’noticing’ (i.e., it focuses the learner’s attention on what he or she does not know, which in turn pushes the learner to try to find solutions), it serves the language learning process through hypothesis testing and serves as a metalinguistic function for language learners. Swain (2000) emphasises the role of collaborative dialogue in which learners are engaged in problem solving and knowledge building. She further argues that dialogue is a joint activity which draws learners’ attention to problems and enables them to verbalise solutions and test out their hypotheses about the L2. This verbalisation provides an object to reflect on and has a positive effect on L2 learning:
As a tool, dialogue serves second language learning by mediating its own construction, and the construction of knowledge about itself. Internalization of process and knowledge is facilitated by their initial appearance in external speech. (Swain in Lantolf 2000: 112)
12From this we hypothesise that a task-based approach will enable the learner to take an active part in the construction of his or her own knowledge and to interact with other learners as outlined by Ellis (2003: 178):
“[T]asks can be seen as tools for constructing collaborative acts”. In this approach tasks are associated with “real world activities” and “real world processes of language use”.
13Tasks can be categorised into two types (Demaizière & Narcy-Combes 2005):
- macro-tasks which are meaningful real-life activities in line with the learners’ future professional environment (constructivist paradigm);
- micro-tasks which are exercises that will enable students to engage in reflective and metacognitive activities, and to practise (cognitivist paradigm).
14It follows that macro-tasks are collaborative activities carried out in teams or groups, while micro-tasks are individual activities. Although the two categories belong to two different and seemingly opposing paradigms, we hold that macro-tasks and micro-tasks are complementary, as they enable the learners to engage in meaningful social activities on the one hand, while reflecting on the language and practising individually on the other hand (Narcy-Combes J.-P. & M.-F. Narcy-Combes 2007).
15Three types of data were collected for the evaluation: a pre-test and post-test were administered to gather data pertaining to students’ language levels, language use was observed in class to identify episodes of language learning and finally, students’ attitudes and views about the blended learning programme were gathered by administering a questionnaire.
16A pre-test was implemented in the first week of the semester, before students started using the new blended learning system, to enable us to investigate students’ levels in English. The communication task selected and developed for the purpose of this study was picture description in the form of a written production. The selection of this task was motivated by the assumption that it is more open-ended linguistically (Swain & Chalhoub-Deville in Bygate et al. 2001) enabling students to be more creative and encouraging learner output. Data were collected from eleven classes comprising 379 students, all non native speakers of English. The post-test followed the same format and was administered during the last week of the course to nine of the original eleven classes following the refusal of two teachers to participate in the post-test evaluation programme. The following study, therefore, concerns the nine classes which undertook the pre-test and the post-test enabling a comparison of students’ scores achieved at the beginning and the end of the course. While 290 students took part in the pre-test, only 159 students were present and completed the post-test, so a loss of 131 students was recorded between the pre- and post-tests. This loss can be explained by a high level of absenteeism (52 students or 40%) and a high drop-out rate (79 students or 60%). Further analysis of the language levels of students who dropped out shows that A1/A2 and B1 levels account for 86% of the total (see appendix 1).
17The resulting written productions were analysed for assessment purposes using a grid based on the Common European Framework of Reference for Languages (CEFRL) incorporating its six common reference levels, criteria and scaled descriptors associated with linguistic competence. The six proficiency levels are arranged in three bands: A1 and A2 (basic user); B1 and B2 (independent user); C1 and C2 (proficient user) (see appendix 2).
18A strategy of “investigator triangulation” (Dörnyei 2007: 58) was employed to ensure descriptive validity of the results by using multiple teachers to collect and interpret the data: six teachers were involved in the evaluation of the productions. To further ensure coherence in the evaluation process a moderation meeting was held to verify that there was broad agreement on the interpretation of level descriptors and rating categories (McNamara 2000).
19Regarding classroom observation procedures, nine classes were observed for a total period of 22 hours of which four classes (about twelve hours) were audio recorded and transcribed. The researcher was minimally involved in the classroom setting and can therefore be described as a “non participant observer” (Dörnyei 2007: 179). During the observations students were assigned a number according to their seating arrangement in class to identify each conversational turn and protect their identity (S1, S2, S3, etc.). Hesitations and speech meanders were kept so that the final transcription reflected accurately what was recorded during the in class task activity. The data were then analysed and examined specifically for examples of “noticing”, hypothesis testing episodes (HTE) and collaborative dialogue episodes (McAllister 2009).
20A questionnaire to evaluate students’ satisfaction of the blended learning system was developed following a series of ten exploratory interviews with students to identify key themes (De Singly 2005). Interviews were single sessions and followed a structured format. The questionnaire (see appendix 3) contains a majority of closed questions with an open question posed at the end in order to obtain qualitative information. It was administered in class during the last week of the semester and was treated anonymously. 158 questionnaires were collected. Likert scales were used to calculate mean scores: each response option relating to the closed questions was assigned a number for scoring purposes (1 to 4). Response data were then entered into the Moodle platform which automatically calculated the mean scores for items addressing the same target.
21This section presents the results derived from the pre-test and post-test, class observations and the questionnaire survey relating to students’ perceptions of the blended language learning programme.
22The French government stipulates that students entering university after secondary school should attain the B2 level in the L2. The results from this study show that:
231/ When students enter university after secondary school, they are mostly B1 or even A2, and only 23% are B2, the level they are supposed to have reached. This paves the way for dissatisfaction and frustration, for if students are A1 or A2 they simply cannot succeed, especially if the tasks they have to perform are level B2 or C1.
Table 1. Pre-test results for 159 students: evaluation of linguistic skills
CEFR level
|
Number of students
|
Percentage of total sample
|
A1
|
5
|
3%
|
A2
|
42
|
26%
|
B1
|
68
|
43%
|
B2 (target level)
|
36
|
23%
|
C1/C2
|
8
|
5%
|
Table 2. Post-test results for 159 students: evaluation of linguistic skills
CEFR level
|
Number of students
|
Percentage of total sample
|
A1
|
1
|
1%
|
A2
|
26
|
16%
|
B1
|
65
|
41%
|
B2 (target level)
|
55
|
34.5%
|
C1/C2
|
12
|
7.5%
|
242/ The blended learning system enabled some of the students to improve: analysis of the individual test results showed that a total of 56 students out of 159 (35% of the sample) improved their linguistic level. Two samples of students’ pre- and post-tests are provided and analysed in appendix 4, together with the assessment criteria grid used (appendix 5). The students’ production improved both quantitatively and qualitatively as they managed to produce longer, richer, more adequate as well as correct and better structured texts. There are more students at the B2 and C1 levels than for the pre-test. Indeed, the biggest change concerns the B2 segment which increased from 23% to 34.5% of the student population surveyed, while the B1 segment remained stable. However, it is difficult to account for the reasons why this is so, although one explanation may be linked to the students’ involvement (see below).
253/ Fewer students are A1 and A2 levels: this segment declined from 29% of the population to 17%.
264/ 290 students from nine different classes participated in the pre-tests but only 159 students were present for the post-test, which represents a loss of 131 students. One of the hypotheses was that students at lower linguistic levels would probably drop out quickly with the blended learning system. This seems to be confirmed by the analysis of the profile of students who had dropped out before the post-test: the majority were lower level students: A1/A2 level (49%) or B1 level (37%) (see appendix 1). Student involvement is another possible explanation.
27Statistics generated automatically by the Moodle platform provide an indication of platform usage patterns: a total of 375 students accessed the lesson space at least once during the period of study representing just 60% of the total number of students. Furthermore, the less advanced students were also the ones who did not log on: 21% of students at the A1 level and 27% of students at the A2 never logged onto the lesson space during the semester compared to 13% of B2 and C1 levels (McAllister 2009). This seems to imply that the system is better adapted to more advanced students, and that the less advanced need the presence of the teacher more, as confirmed by the analysis of the results of the questionnaire.
28The self-training centre was not exploited by students: only 189 students actually registered for this part of the programme (registration was not automatic), of which only 129 used the system, representing just under 20% of the total student population (i.e., of the 660 students enrolled in first year).
29In line with Chapelle’s view (2001: 32) this study focuses on the analysis of learners’ dialogue in order to evaluate the potential of the blended language learning programme for second language acquisition:
A cultural constructivist approach hypothesizes that the experience crucial for individual cognitive development takes place through interaction with others, and therefore key evidence for the quality of a learning activity should be found in the discourse that occurs in the collaborative environment.
30The following transcription extracts provide evidence of language learning in face-to-face tutorial classes. The following short extract taken from one class shows a total of five HTEs as well as two collaborative dialogues between students S2 and S1 and S7 and S1. The teacher (T) and students S1 and S7 both provide scaffolding support to student S1 to help him produce well-formed output.
Extract 1 from classroom observations
S2: It depends of our budget
T: It depends on
S2: It depends of
S1: no, she said it’s “depends on” not “of”
S2: It depends on, how you say “état d’esprit”?
S7: state of mind
S2: state of mind, Je ne me rappelle plus la question...
T: Would you buy a counterfeit product?
S2: Personally, I don’t think
T: I don’t think so
S2: It’s personal. It depends on the “centre d’intérêt”
T: So it’s when you really want a product. I won’t mention Rolex anymore.
(Students laugh)
T: If it’s a product which looks like the real thing, but it’s much cheaper, would you buy it?
S7: It depends on the product. It’s very important to know how it is producted.
31A second extract illustrates how a learner experiments with hypotheses to see what sounds better, but the ill-formed output was challenged by teacher corrective feedback which resulted in the student modifying his output.
Extract 2 from classroom observations
T: So what’s the effect of people buying less on eBay?
S12: eBay wins, gains less
T: earns less
S12: eBay earns less
32These examples provide validation for Swain’s claims that producing output is one way of testing a hypothesis about linguistic correctness or comprehensibility and can have a positive effect on second language acquisition.
33The third part of this study concerns the students’ attitudes towards and perceptions of the blended language learning system. Students were first asked to rate how much they believed they had progressed across six skills during the semester. Answer options ranging from regression through to positive progression were attributed a score between 1 and 4 and entered into the Moodle platform. The platform then automatically calculated the mean response score as shown in Figure 1.
34The Moodle extract shows that the highest mean score is attributed to oral comprehension and oral production (2.9 out of 4) highlighting the key areas where students believe they have most progressed most, while the lowest score is attributed to the mastery of grammar (2.3.), i.e., the area where they believe they have progressed least.
Figure 1. Moodle extract
35We then asked the students what criteria they thought were the most important for learning a language. The list of criteria used for this question was based on feedback provided by students during the exploratory interviews and thus entirely reflected their views. The mean scores calculated by the Moodle platform are shown in Table 3.
Table 3. Importance of criteria
Rank
|
Criteria
|
Mean score (on a scale of 1 to 4)
|
1
|
Oral practice
|
3.7
|
2
|
Small groups
|
3.5
|
3
|
Feedback
|
3.4
|
4
|
Customised exchanges with teacher
|
3.3
|
4
|
Learning grammar
|
3.3
|
4
|
Individual work
|
3.3
|
4
|
Interactions
|
3.3
|
5
|
Class time devoted to L2
|
3.2
|
6
|
Real-world tasks
|
2.9
|
7
|
Team working
|
2.6
|
36Students were then asked to evaluate the system according to the same criteria. Table 4 presents these results. In evaluating the blended learning system students pointed to small groups, feedback and interactions as major advantages, which were also seen to be some of the most important criteria to them. However, students noted some dissatisfaction with oral practice which seems in contradiction with the increased interactions in class. Another aspect of the blended learning model viewed favourably by students, but not rated as being important to them, is the real world tasks.
Table 4. Satisfaction rating
Rank
|
Criteria
|
Mean score (on a scale of 1 to 4)
|
1
|
Small groups
|
3.1
|
2
|
Interactions
|
3.1
|
3
|
Feedback
|
3
|
3
|
Real-world tasks
|
3
|
4
|
Oral practice
|
2.9
|
4
|
Customised exchanges with teacher
|
2.9
|
4
|
Team working
|
2.9
|
4
|
Individual work
|
2.9
|
5
|
Class time devoted to L2
|
2.4
|
6
|
Learning grammar
|
2.3
|
37The student responses also identified a number of problem areas which require our attention:
38- oral practice: the questionnaire responses show this is important but students were dissatisfied (2.9);
39- time devoted to L2: the reduction in face-to-face time with the teacher inherent in the blended learning system is viewed negatively by students (2.4);
40- team working: this is viewed as unimportant to the students (it is ranked last on the criteria list), which is in contradiction with their future professional role in the workplace where they will have to work in teams. They also express dissatisfaction with this element of the blended learning system (2.9). As we have seen, group work is a key element of the collaborative learning environment promoted by the blended learning system and is essential for second language acquisition. Students do assignments and projects in groups, but group work and team working represent a stumbling block for them. 61% of respondents claim to have had difficulties meeting up with other students and 29% had difficulties maintaining team cohesion.
41- preparatory work: in response to the question “Did you do the preparatory work of each task?” only 22% of students admitted doing all the preparatory work required for each task. The majority (72%) admitted to doing it only partially or irregularly. Concerning the time devoted to preparatory work, 39% of respondees spent 1 to 2 hours on it, a further 28% spent 2-3 hours, but 16% spent less than 1 hour.
42Analysis of the open-ended question highlights some of the same issues particularly relating to face-to-face instruction and team working. Students were asked to complete the following phrase: “For you, learning English autonomously as you have experienced this term represents...” 114 students responded to this question. A first analysis helped define categories for the classification of student responses: just under half of the responses concerned positive aspects associated with the blended learning programme, while a third highlighted negative aspects or problems with the blended learning programme:
Table 5. Student perceptions of the blended language learning programme
Strengths of the blended learning programme (51 responses)
|
- ability to work at our own pace (3)
|
- increased responsibility for our own learning and progression (8)
|
- greater autonomy (11)
|
- improved organisational skills (6)
|
- improved time management skills (2)
|
- greater personal investment in tasks (6)
|
- real life learning situations (2)
|
- personalised feedback from the teacher contributed to progression (3)
|
- group work fostered different opinions and ideas and contributed to progression (10)
|
Problems linked with the blended learning programme (37 responses)
|
- difficulty in assuming responsibility for one’s own learning (5)
|
- difficulty in working autonomously (3)
|
- difficulty in organising work schedule (1)
|
- too much personal investment required (3)
|
- insufficient number of hours of face-to-face instruction (17)
|
- difficulty in managing group work and/or team members (8)
|
- 1 “a teacher is essential”, “a class should be with a teacher”, “it is very difficult to follow witho (...)
43Table 5 presents the key themes verbalised by students organised into two categories, whether they pertain to strengths or problems linked with the blended learning programme. The number of occurrences is indicated between brackets. What emerges is that a number of themes are recurring in both categories: responsibility, autonomy, organisation skills, personal investment and group work. On the one hand these are perceived positively by some students leading to progress and personal development, on the other hand they are perceived by others as constraints generating difficulties and problems and sometimes leading to demotivation. It is striking that 19 responses mentioned greater autonomy and greater personal responsibility for learning as strengths of the system demonstrating an increased sense of learner autonomy. In contrast, the idea of feeling “alone” was mentioned in nine different responses and the lack of face-to-face instruction was highlighted by students as a major criticism of the programme (it was mentioned 17 times). Students’ comments also serve to highlight this issue: “un professeur est essentiel”, “un cours, c’est avec un professeur”, “cela est très dur à suivre sans la présence d’un professeur”, etc.1
44The students we have to deal with are the students who come to the university, not ideal ones. Students’ profiles, representations and beliefs should be taken into account when innovation is implemented. Traditional educational practices may be boring to some students, but novelty appears threatening to most of them in the context of this study. Of course, it does not mean that no change should take place, but the process will take time and empathy is needed.
- 2 A national strike was started by university lecturers-researchers protesting against government ref (...)
45In a blended learning system, distance works as a filter: the less adapted students are the most likely to drop out quickly. The exam results showed very little change in the grades between the first and the second semester despite a lecturer-researcher strike,2 but a hundred students had dropped out (appendix 1). So, if the system is implemented to reduce the drop out rate in first year, then it is not the correct answer to solve the problem. The answer for the less advanced students seems to be to maintain the number of teaching hours, also to reduce the number of students to 18-20 per group and provide technological support as a complement.
46We have validated the hypothesis that the system works better for L2 acquisition, however time is needed for students to appropriate the system. However theoretically sound the system may be, it cannot be forced upon learners when they are simply not ready for it. This may explain why so many innovative experiments fail.
47These qualitative interviews and questionnaires will enable us to make a number of changes to the blended learning system:
48- the pre- and post-test results have enabled us to pinpoint the most frequent problems which will in turn feed into the development process of the self-training centre;
49- development of new tasks and revision and simplification of task instructions and preparatory work;
50- resolution of technical problems;
51- creation of user guidelines to better inform students how to access and use the system.
52The aim of this paper was to assess the outcomes of our project halfway and to gather evidence that students’ learning had occurred. Obviously, we have not been able to draw straightforward conclusions, let alone reach absolute truths, which we did not expect to do anyway. The results show measurable effects of the blended learning system on student learning concerning its potential for second language acquisition as demonstrated by post-test results and classroom language use. However, only 60% of students engaged with the on-line platform and even fewer used the self-training centre, so it is difficult to account for these improvements. We still have to imagine solutions to the problem of team working and find a way to reassure the students that they have more teacher contact in a blended learning environment than they would in a group of 50 or 60. However, qualitative methods have enabled us to identify key areas of improvement for the blended language learning system and to be able to better understand students’ attitudes and behaviour.