Evaluating and Revising Courses from web resources educational

Danièle HERIN, Michel SALA, Pierre POMPIDOR

Abstract. The World Wide Web offers a great availability of heterogeneous educational resources. This suggests the idea that such materials can be re-used in compose courses. In this paper we address this issue by proposing an architecture for composing teaching courses using "the best parts" of heterogeneous educational materials available on the Web. Course composition relies on a simple but effective evaluation methodology which reproduces real techniques used by teachers in composing and improving classroom courses. The final goal of this article is to help the teacher to construct his course until the obtension of a steady course.We present our initial work and discuss about future developments.

Nowadays, the WWW (World Wide Web) constitutes the biggest source of educational courses of the globe. Following different approaches, many efforts have been done towards providing learners with courses able to teach without the help of a real teacher. From the simplest courses which appear as a sequence of web-pages displaying educational contents, to the more sophisticated ones, provided by educational applications which try to support the learner during the learning process in a more or less intelligent way, respectively ITSs (Intelligent Tutoring Systems) or CAL (Computer Assisted Learning). Among all courses teaching the same subject and developed according to different computer based learning educational paradigms, there are not any absolutely better than others: each of them is able to teach better than the others certain parts of the course. This can be due to the characteristics of the particular approach used or simply because the course is better designed or because it presents higher quality contents.

So far, most of teaching courses built according to each different educational paradigm have been composed using didactic material expressly designed or adapted for them. To the first category belong courses available as a sequence of web-pages and most of CAL applications or ITS [1]; in other cases, teaching materials are produced by modifying existing ones [3]. Other applications allow building instructional courses from already existent educational materials [2,4], but re-used materials for composing the course are of the same type, i.e. homogeneous (e.g. all HTML-files, all CAL materials). Using homogeneous didactic materials for composing courses guarantees semantic coherence between the different parts composing the course itself. Moreover, when didactic material is designed explicitly for a given course the quality of the teaching material is not put in discussion.

We focus our attention on the problem of re-using existent on-line educational materials developed according to different instructional paradigms and of composing the best parts of them in order to obtain high-level teaching courses which combine all best characteristics of different approaches. We propose a methodology and an architecture which constitute a support for teachers in building educational courses using "the best parts" of heterogeneous educational materials available on the WWW. Teacher is helped in detecting which parts of existing educational materials are the best to use for composing teaching courses and in checking that among different parts composing the built course holds semantic coherence even if they come from heterogeneous sources. This is done by proposing each composed course to students and by evaluating its effectiveness on the basis of comprehension the students have of contents of the course. Starting from such evaluation, learning is made on the quality of the course and feedback is provided for revising and improving it. In particular, the evaluation of students’ reaction to the proposed course gives advice on both the quality of single parts composing it and on effectiveness of the whole course, i.e. it reveals if the order of the parts of the course (curriculum) is a good one and if semantic coherence holds among them. Learning, in our approach, relies on a pragmatic methodology of evaluation and revision of built courses, which reproduces the real way according to which teachers compose and improve traditional classroom courses.

In this paper we describe our initial efforts in designing an architecture responding to the characteristics described above. We also propose a first version of the evaluation methodology that constitutes the main part of our work. We present successively the proposed approach, the conceptual structures and the evaluation methodology. There were already works on the domain from P. Brusilovsky [5,6] on the Web-Based Education (WBE).

The overall process is characterised by the following steps:

1. Consensual contents of the course: contents of a teaching course generally constitute a consensus for what concerns the main concepts to teach in the course. For example, a teaching course on programming languages is accepted to be composed of two main parts: data structures and algorithmic structures. In many cases, a concept constitutes a prerequisite for another concept: in this case, we say that the two concepts are related.
1. Use of different textbooks: when a teacher builds a course, he consults different books or articles treating the subject of the course. In general, a teaching course is built by composing pieces of knowledge selected from different textbooks.
2. Content planning: each teacher has his own teaching method according to which he decides about relevant concepts to teach in the course and their teaching order. We call curriculum of a teaching course the ordered sequence of concepts taught in a course. Each teacher sequences his own curriculum.
3. Composition of the course: once a curriculum has been sequenced, associating each concept in it with pieces of knowledge selected from different textbooks substantiates it.
4. Teaching session: the composed course is presented to students.
5. Course evaluation by testing students: teacher evaluates effectiveness of his course by examining how students perform on quizzes and tests on all concepts composing the teaching course. In particular, the teacher detects which are typical errors made by students on the different parts of the course.

We propose an architecture (figure 1) and an evaluation methodology based on an analogy with observations listed above:

1 Global Ontology (GO) describes consensual contents of the course: we dispose of a consensual conceptual structure, GO, which represents the consensual knowledge about a given teaching course. It contains all concepts composing the course as well as prerequisites holding among them. Concepts in GO are called Educational Units (EUs). Prerequisites among EUs are represented by precedence constraints.

2. Local Ontologies (LO) describe different didactic tools: on-line educational materials, called didactic tools replace traditional textbooks. The teacher disposes of a great deal of didactic tools available on the Web treating the same subject of a given teaching course. A didactic tool teaches a set of EUs. Each didactic tool is described by a conceptual structure called LO containing all EUs taught by the didactic tool.

3. Curriculum Planner to help teacher in sequencing the curriculum: the teacher sequences his own curriculum and the Curriculum Planner checks its consistency with respect to GO. A consistent curriculum is a sequence of EUs present in GO respecting all precedence constraints.

4. Course Composer to compose the course: the Course Composer composes the course by associating curriculum EUs with the parts of didactic tools able to teach them. This is done by instantiating each EU in the curriculum with one of the corresponding EUs contained in LO.

5. Teaching Session: during the Teaching Session the course is taught to students.

6. Evaluation methodology: the teacher makes the evaluation of the effectiveness of a teaching course by testing students on all EUs composing the curriculum. For each tested EU he gives a teaching coefficient indicating how well the EU has been "learned" by students and detects errors made on it. The evaluated course is stored in the Curricula Database.

In the setting of this paper, we won't take in account the problems of homogeneity of the different sources. On the other hand this homogenization is necessary to construct a curriculum. We consider solely that the GO is defined by the teacher and that the LO of every source is gotten while using a parser [7].

Fig. 1 General architecture

An example of Global Ontology GO1 of a course teaching the subject A is denoted as follows:

GO1 = {A(B[er1],C,D[er2,er3]), {B ® C}}

where GO1 identifies a particular Global Ontology; A is a composed EU; A(B,C,D) means that A is composed of B, C, and D; B, C and D are elementary EUs; er1, er2 and er3 are errors and B[er1] means that error er1 is relative to B; {B® C} is the set of prerequisites holding among EUs composing GO1 and B ® C means that between B and C holds a precedence constraint.

We can think at GO as a textbook teaching the subject A where A is the whole book, B and C are chapters; the precedence constraint that holds among B and C means that chapter B teaches concepts which are necessary for understanding contents of chapter C.

The GO of a teaching course is built consensually by many teachers. We consider it as given. It is used as a scheme of reference to generate curricula of the course.

Didactic tool is the name given to denote heterogeneous educational materials available on the Web. Each didactic tool is an entity that has been developed autonomously according to different instructional paradigms and for different purposes. It can be, for example, a text, a picture, CAL materials or an educational software application. A didactic tool teaches a set of EUs on a given domain. EUs taught by a didactic tool correspond to the smaller unit that can be identified by an http-address into which the tool can be decomposed. For example, while sections teaching different concepts can be recognized and isolated in a text, it could not be possible to decompose in sub-parts a software application presenting a course on, for example, "Programming Languages". In this case we can only say that the software application teaches the EU "Programming Languages". We use EUs coming from different didactic tools for composing teaching courses. Because of the heterogeneity of such materials we need to define a common representation for describing contents taught by different didactic tools. Contents of a didactic tool are described by a LO.

Local Ontology of a didactic tool is represented as a hierarchy of aggregation whose nodes represents EUs taught by the tool. Given the GO of a teaching course, LO of a didactic tool is built using GO as scheme of reference. For each EU appearing in GO, the same EU is searched in the didactic tool; found EUs are then organized respecting the hierarchy indicated in GO. The resulting LO constitutes a subset of the GO of reference.To each EU composing LO is associated the following additional information:

Each EU composing the LO is associated with errors made by students in learning the EU. B[erj] indicates that error erj with j Î N is associated to EU B. For each detected error is also indicated the percentage of students that have made it. In the notation B[erj_p] the parameter p represents the percentage of students that have made erj associated to the EU B. The percentage p is calculated by the following

p = (e/s)*100 (1)

where s is the number of students that have learned B and e is the number of students that have made error erj.

For each EU in Local Ontology a teaching coefficient indicates how well the didactic tool teaches the EU. In the notation B_x x represents the teaching coefficient of EU B; it is calculated according to the following equation:

notation B_x x represents the teaching coefficient of EU B; it is calculated according to the following equation:

x = (n/s) where n = å _i=1..s n_i (2)

where s is the number of students that have been taught with B and n_i is a score indicating how well B has been learned by student i. The score is given with respect to a parameter of reference r and depends on the teacher t who has given the evaluation. An example of score indicating how well a student has learned a given EU could be 5/10 where n_i = 5 and r = 10.

The following example of Local Ontology

LO1 = {A₉(B₈[er1_60%],C₉,D₈[er2_80%,er3_50%])}

describes the Local Ontology LO1 of the didactic tool 1 which teach the EUs A, B, C, D where B₈ indicates that the EU B has teaching coefficient x = 8 (for r=10); B[er1] means that error er1 is relative to B and er1_60% is the percentage p = 60% of students that have made error er1.

Considering the above GO1, the following examples of Local Ontologies are relative to didactic tools 1, 2 and 3 respectively

LO1 = {A₉(B₈[er1_60%],C₉,D₈[er2_80%,er3_50%])}

LO2 = {A_8.5(B_7.5[er1_80%],C_9.5,D_8.5[er3_50%])}

LO3 = {A₇} ......

The construction of LO can be made semi-automatically by a parser (it is not the objective of this paper) which tries to individuate in the didactic tool the block of information concerning a particular EU. Such block of information is identified by a http-address. To each EU composing a LO is associated the http-address that corresponds to the physical location on the Web where educational material teaching that EU is located.

The Course Generator is the component that helps the teacher in sequencing the curriculum of the course and that creates the course by substantiating the concepts in the curriculum with educational materials able to teach them. Each of these tasks is carried out by one of the following components:

The first step in building a teaching course is sequencing the curriculum, i.e. deciding which EUs are to teach in the course and their teaching order. Making reference to the Global Ontology which describes the teaching course, the teacher t sequences his own curriculum by choosing and ordering EUs he wants to teach in the course. He also indicates which are errors associated to EUs in GO that he wants to minimize (for example, making reference to GO, the teacher can choose that he wants to build a course that minimizes er3 on D). Then is invoked the Curriculum Planner that checks if teacher’s curriculum has been built respecting all constraints imposed by GO. A curriculum is considered correct if:

• EUs composing the curriculum are a subset of EUs appearing in GO;
• all precedence constraints among concepts indicated in GO are respected.

The Curriculum Planner guides the teacher in composing a correct curriculum by pointing out inconsistencies of the proposed curriculum and suggesting possible solutions. The result of interaction between the teacher and Course Planner is the generation of a correct curriculum that is a sequence of EUs present in Global Ontology and respecting all precedence constraints. We call such a curriculum the "non-instantiated" curriculum of the course.

Making reference to the above Global Ontology GO1, an example of correct curriculum could be:

P 1_(GO1) = <B,C,D>. (3)

P 1_(GO1) identifies is the non-instantiated curriculum of a course teaching the subject described by the Global Ontology GO1; <B,C,D> are the EUs that have to be taught during the course. (3) is a correct curriculum in the sense that it satisfies both the conditions listed above, i.e. EUs B, C, D that compose it constitute a subset of EUs composing GO1 and the precedence constraint among concepts B and C is respected.

Given the non-instantiated curriculum of the course, the Course Composer generates the teaching course by substantiating each EU of the non-instantiated curriculum with one of the corresponding EUs contained in LO. The choice of the EU to associate to a concept in the curriculum is made according to one of the following strategies:

• choosing the EU with the higher teaching coefficient (cf. §5.1). If there are many EUs with the same teaching coefficient, it is chosen the one which minimizes errors associated to the corresponding EU in GO;
• choosing the EU that minimizes errors indicated by the teacher in the phase of course planning.

In this sense, the obtained "instantiated" curriculum is composed of "the best parts" of available didactic tools.

An instantiated version of curriculum P 1_(GO1) with respect to Local Ontologies LO1, LO2, LO3 is the following

p _{(P 1,GO1)}=<B_(LO1,8),C_(LO2,9.5),D_(LO1,8)> (4)

where B_(LO1) indicates that EU B is taken from Local Ontology LO1. B has been chosen from the Local Ontology with the higher teaching coefficient for it:

B_(LO1,8)=max_{teaching coefficient}{B_(LO1,8),B_(LO2,7.5)};

D has been chosen in order to minimize er3, as indicated by the teacher in the previous section:

D_(LO1,8)=min_{percentage er3}{D_(LO1,8)[er3_50%],D_(LO2,8.5)[er3_80%]}.

We present the initial version of the evaluation methodology on which our system is based. We will describe how teaching courses are evaluated on the basis of students’ mastering of course contents and how evaluations are used to improve the quality of courses built in the future.

The instantiated curriculum representing the teaching course is proposed to students and a teaching session takes place. At the end of the teaching session the course is evaluated on the basis of knowledge acquired by students on the subject taught. The teacher who tests students on all EUs composing the curriculum makes course evaluation. For each tested EU, the teacher detects which are errors made on that EU. These errors determine how well the EU has been learned. On the basis of made errors, the teacher also indicates a teaching coefficient that is a numeric value representing the level of mastering gained by students on the EU.

Each EU appearing in the curriculum of a course is associated with errors made by students in learning that EU. The notation B[erj] indicates that error erj with j Î N has been made by students in learning the EU B. Errors detected on EUs can be of different types:

- errors caused by the didactic tool that is not good at teaching the EU: each error represents a misconception of the student on the part of the EU that is not taught well. For example, B[er1] indicates that students have not well learned the part of B corresponding to er1.

- errors given by wrong sequencing of the curriculum: the EU has not well understood because it requires a prerequisite EU that have not been put in the curriculum. In this case, prerequisites among EUs in the curriculum are not respected.

- errors propagated between related EUs: if a EU is prerequisite for another EU making an error on the first EU causes bad understanding of its related EU.

Errors detected on EUs are indicated in the evaluated curriculum of the course.

For each detected error is also indicated the percentage of students that have made it. In the notation B[erj_p] the parameter p represents the percentage of students that have made erj associated to the EU B. If s is the number of students that have learned B and e is the number of students that have made error erj, the percentage p is calculated using equation (1).

• Teaching coefficients

Teaching coefficients are associated to each tested EU: they indicate how well a EU is learned by students. In the notation B_x x represents the teaching coefficient of EU B. Let s be the number of students that have been quizzed on B and n_i with i=1..s be a score indicating how well B is learned by student i, which depends on a reference r and on a teacher t; the teaching coefficient x relative to the EU B is calculated using equation (2).

Results of evaluation of a course are stored in the instantiated curriculum of the course. Let’s consider (4); its evaluation given by a teacher t on a class k of students is the following

p _{(P 1,GO1)}=<B_(LO1,8.5)[er1_55%],C_(LO2,8)[er4_80%],D_(LO1,8.5)[er2_65%,er3_45%]> (5)

where B[er1] means that error er1 is made on the EU B; er1_55% indicates the percentage p=55% of students that have made error er1 which is calculated using equation (1); in B_(LO1,8.5), x = 8.5 is the teaching coefficient representing students’ comprehension of B which is calculated using equation (2). The evaluated instantiated curriculum of the course is stored in Curricula Database.

According to the new evaluations of EUs in the course, Local Ontologies are updated: teaching coefficients and percentages of errors given for EUs in the curriculum are used for re-calculating teaching coefficients and error percentages of the corresponding EUs in Local Ontologies. Let y=(m/k) with m=å _i=1..k m_i be the teaching coefficient representing the evaluation given by the teacher for a EU and let x=(n/s) with n=å _i=1..s n_i be the teaching coefficient of the in Local Ontology to which it refers. The new teaching coefficient z on this Local Ontology is obtained using the following equation:

On the contrary, let p=(d/k)*100 be the percentage of students that make a particular error on a EU in the course and let q=(e/s)*100 be the percentage of the same error on the corresponding EU in Local Ontology. The new percentage w on Local Ontology is calculated using the following equation:

Next time curriculum P 1_(GO1) is instantiated, the association of EUs in the curriculum with EUs in Local Ontologies LO1, LO2 and LO3 will be made according to the modified teaching coefficients and error percentages.

In this paper we proposed an approach which is pragmatic and involves strongly the teacher. It reposes on the fact that course evaluation and, indirectly, the evaluation of the quality of used educational materials is completely left to the free will of the teacher (even if we check coherence among evaluations given on the same course by different teachers). We only point out if students have "well passed or not" the course. In this sense we help "mass teaching", like that of collective teaching made in classrooms, instead of individualized teaching. Our goal is not that of determining emotional or psychological implications that can cause bad understanding of certain parts of the course; we leave these problems to specialists in the domain of education.

We have also described how the domain knowledge of the course, i.e. the GO, is modified on the basis of successive teaching experiences by adding relevant information about the teaching. We are interested in studying more deeply how GO can be widen in order to complete the knowledge it expresses. In order to do that, we would reject the hypothesis made in this paper that LO and curriculum of the course are composed by a subset of EUs composing GO: in fact, we have supposed the only knowledge to which make reference is the consensual one. In the reality, on the contrary, didactic tools are able to teach more EUs other than that ones contained in GO. If we accept this hypothesis, we can learn how to complete GO by learning about EUs taught by didactic tools and present in curricula sequenced by the teacher.

Future work will consist in revising the teaching course which relies on the analysis of errors made by students who have been taught with that course. Error analysis makes it possible to recognize the type of errors made and, as a consequence, to determine the cause that generates them. It allows deciding about actions to take in order to improve course effectiveness. The Revision Module on the basis of evaluated courses stored in the Curricula Database makes error analysis. It recognizes the types of the errors associated to the different EUs composing the course and modifies LO and GO in order to improve the future Course Planner and Course Composer work. Another goal with consist in developing a prototype for validating our evaluation and revision methodology. We also intend to investigate how machine learning techniques can be applied for improving their effectiveness and how they can be used in order to provide more intelligent support to the teacher in planning his curriculum.