The Italian experience on evaluation systems of learning results in e-learning environments can be catalogued according to the following access methods:

a. In a virtual class, or rather single individuals that have subscribed to a course and have used web based training products

b. In dedicated environments called Learning Points in which it is possible various forms of ICT in an integrated way

c. In videoconference settings that can refer to and use virtual classes/environments; or can refer to class- laboratory type settings. This second case is represented by “business television”.

To better understand how the assessment and evaluation of results systems vary in relation to the three ways described above, the report that follows will be conveyed in the following points:

• Description of the conditions and technologies that characterise the learning environment

• Didactic methodology predominantly used

• Possible control learning system in relation to the learning environment

• Reaction of the participants to ICT technologies and to the learning assessment and evaluation system used

• Main experiences and references

Following the experiences referring to the three ways mentioned, the current tendencies will be presented with particular reference to the socio-cultural aspects that characterise the evolution of the use of e-learning solutions.

First case

Virtual class, or rather single individuals that are subscribed to the course and access web based training products

• Description of the conditions and technologies that characterise the learning environment

This is the most common situation, commonly called Distance Learning where the training user can consult a training offer, commonly consisting of modules and/or didactic units accessible online through internet networks or through an intranet.

The interaction between the user and the informative system can be:

o A free type, in this case the user accesses a catalogue published by a content provider or a public/private training organisation, chooses the training products that he/she likes and if he/she decides to access the course system administrator allows him/her to access the courses, assigning them a login and password. Delivery can be free or paid. In this case the choice of the user can be supported by course format that illustrates: the learning objectives, the entry prerequisites possibly connected to a questionnaire that helps the user to verify if they have the competences or not, the methods of completion, the possible assistance of a tutor and access to frequently asked questions and forum, the method and times of completion, the method of checking results

o An organised type, this is the typical case of courses managed not directly by the user but by companies. In this case the training needs analysis of the user or users is carried out by the Human resource management and development officers, the type of course is often chosen by the company and can foresee the attendance of courses using standard training or commercial products, or training products that the company have had created. In this second case the cost of subscription and attendance is largely supported by the worker’s organisation that therefore has a specific interest in the learning results. Thus, in this case, the learning assistance and assessment systems are generally more accurate and targeted.

• Didactic methodology predominantly used

The didactic methodology takes on in this case notable importance as regards learning results and the critical success factors can be summarised as follows:

o Calibration level in relation to the target population of users – schooling/culture, age, type and level of motivation to learn, spendabiltiy of knowledge/competences acquired both at a personal and working level, etc.

o Level of interaction and type of methodological approach (inductive, deductive, …)

o Level of tutorial assistance provided

o Possibility for further investigation both with documental analysis and with on/off line experts

o Interaction method with verification system to control and to manage own learning

• Possible learning control system in relation to the learning environment

In these cases the learning control system is strongly conditioned by the free or organised mode of access to the training offer described above

o In the first case (free access on the user’s part) the learning control system is predominantly entrusted to the return: the results of exercises that can be included in the learning course and test results that can be used either in interim, or at the end of the learning course. The more recent training web based products include a reasoned return of the student’s answers, or rather they help the student understand why the answer provided is correct or incorrect and in the second case they advise the user to follow a path where they go back to understand the issue or to contact an online expert. In short, this first case implies self-evaluation on the student’s part.

o In the second case beyond the prerogatives described above the learning control system takes into account the following two fundamental aspects:

• Reference to the competence gap that is at the basis of the training needs

• Possibility of transferring the competences or the elements of competences acquired with the training to a work based context

Both aspects cannot be entrusted only to online control but they require real confirmation “in the field”. This type of control occurs through either self-evaluation processes or through evaluation carried out by someone else.

• Reaction of the participants to ICT technology and the learning control and evaluation system used.

For the participant’s reactions (in reference to the Kirckpatrick model), we intend the level of control of learning perception of the user and his/her difficulties to complete the contents (explanatory clarity, etc), to navigate through the learning course, to benefit from support/analysis, to use ICT technologies foreseen.

This type of control presented only in on line training cases connected to training processes structured by significant public and private organisations is infrequent but it is of underlying importance to maintain high levels of motivation to learn to continually improve the quality of training services.

The reaction level is measured through a questionnaire that can be indifferently provided online as well as offline.

For the reasons set out above it is important to return the user’s questionnaire results with a comment on what has emerged.

• Main experiences and references

In Italy, like in the rest of Europe, the first generation Distance Learning experience is rather disappointing. The reasons for this failure can be summarised as follows:

o Difficulty on the user’s part, above all if they were accessing the training offer as single users, to understand how to find themselves the choice of products as their real perception of their appropriate training need was unclear or they were unable to match their training needs and the learning objectives declared in the course description.

o Difficulty to endure in time of the learning course due to diffused incapacity of users to connect and to transfer the learning into their real context.

o Low quality of training products deriving from an inadequate didactic planning; the result was often conditioned by low levels of interaction, reasoning by contents and not by learning objectives (see the transfer on electronic support of traditional textbooks)

o Sparse tutorial assistance and/or no timeliness of expert assistants online.

With the creation, from the years 2002-2003, of second generation Distance Learning and more recently a third generation, one has sought to remedy this disaster that has cost enormous resources above all in terms of either public or private economic resources. One has progressively developed both the awareness or the ability to use ICT technologies in an integrated way including them in blended courses as well as traditional training interventions.

We would now like to mention a series of reviews that support the description of the situation in this paragraph; available documentation linked to this report.

• E-learning guidelines

• Errors to avoid

• Corporate e-learning

• Route from the first to the second generation of e-learning

Second case

Dedicated environments defined Learning Points in which it is possible to use varied ICT technology in an integrated way

• Description of the conditions and technologies that characterise the learning environment

In Italy from the end of the 90s, both at a public and private institution level, a series of experiences were launched to structure learning environments of a high ICT technological impact.

In both cases one intended to solve the problem of user isolation by adopting the tutorial form in presence to help the user not only to focus on his/her appropriate training needs but also to assist him/her in defining the most suitable learning course and the use of ICT technologies not yet all familiar.

Among the most significant experiences we would like to mention:

o As regards public organisations, the Tuscany Region experiences with the “TRIO” project aimed at a broad public of private users and that of INDIRE (Florence) with the creation of a more virtual than physical environment to train the large population of teachers in state schools

o As regards private organisations the ISVOR FIAT experience with the production of Learning Centres and Learning Points originally designed for self training of the staff of the FIAT Group company (Ferrari. Fiat Auto, Gesco, Fiat Avio, …) then exporting the experience to funded projects intending to develop specific territorial areas

• Didactic methodology predominantly used

The success of the experiences mentioned above is directly proportional to the project strategy and training engineering effort. We would now like to give you a brief description of the approaches and methodological solutions adopted in the three cases.

o TRIO project

The solution adopted included the production in Tuscany of a network of Centres; or rather physical places provided for more advanced ICT technology such as support to a broad and articulate training offer catalogue, with carefully prepared web based training products from a communication and graphical presentation point of view.

These products were structured to transfer fundamental elements of knowledge supporting learning also through the use of teachers and testimonies of excellence produced prevalently in videoconferences.

The tutorial assistance in presence served to help the user to understand the objectives and the content of the training offer and to help the students in the use of ICT technologies available in the Centres.

The limits of this initiative was the fact that the users that had benefited from this training method had remarkable difficulties in transferring the knowledge learnt to the workplace reconfirming that the Knowledge based approach adopted as a training strategy did not appear to the students to be substantially different to that commonly adopted by traditional schools. This fact led to the progressive dimming of interest in the project and the Centre’s decrease in popularity.

o INDIRE project

The solution adopted by the INDIRE, an organisation at the service of the Public Administration to train teachers, assumed an approach strategy to training, the need not only for knowledge but also for know how, or rather to immediately apply what has been learnt in a daily context.

The INDIRE training offer even if apparently structured to a planned catalogue, foresees and stimulates users both in processes of assimilation through the individual use of education, and the exchange of experience and the practical application of what has been learnt in the logic of a professional family.

The success of the initiative is decreed by the thousands of subscribers that continue to benefit from the training services.

The didactic solution foresees an individual completion of online training products organised in learning courses to solve problems.

The type of tutorial assistance is fundamentally online.

o Learning Center/ Learning Point project developed by ISVOR Fiat

The solution adopted using in a broad and integrated way, ICT technologies available in a physical place, in an identical way to the TRIO project, starts from a more systematic approach that is of processes and competences.

In other words, a user that goes to the Learning Point (LP) welcomed by a tutor that helps him/her to focus his/her needs of creating a product/service or to solve a problem.

Once they have identified and shared the result to be reached as well as the way to reach and measure the quali-quantitative value, the tutor, through methodologies and tools which are by now consolidated, undertakes a balance between the expected and necessary competences to reach the result and those which the user may already possess.

Having thus detected the competency gaps, or rather the user’s personal training need, the tutor defines and shares, in a sort of training contract the learning course that will lead the user to achieve the expected result.

This learning route can generally be represented as a series of loops through which, with didactic strategies that are sometimes inductive and sometimes deductive, lead the user to immediately apply the cognitive, operative/applicative and behavioural learning in achieving the “final learning object”. The didactic material can be more disparate: from web based products to manuals, from film clips to leaflets and technical reports, always used in a targeted and functional way to achieve the result.

The learning course generally foresees 15%-25% of the learning time interacting individually and/or in small groups with experts that are either physically present or online.

In this course ICT is not in the forefront but is used in an integrated and functional way to achieve the result; thus for example to achieve the final result the user can access, using the videoconference tool together with a software to share files, a series of experts that will help him /her to identify and resolve difficulties connected to the exercise of specific competences.

The tutor, even though not an expert in the different disciplines, is always present and carries out a didactic support action, intervening and possibly modifying the learning course on the basis of the individual user’s difficulties, maintaining the objective that needs to be fulfilled.

The success of this initiative is given by the fact that the user has tangible demonstrations of having entered the Learning Point with a problem and of leaving not only with the problem solved but also having acquired the competences of the process and product that have guaranteed the result.

This experience is being transferred from the productive context of FIAT to a more general development of territorial areas such as, for example the LP network created in the Tuscany Region between Lucca and Pistoia for the basic and specialist training of staff working in the paper and shoe industry. This is also true for the LP of Lanusei in the Sardinia region aiming to decrease the isolation of citizens and of the productive and tourist reality of small towns in the mountains; or still the LP of Piossasco to sustain the structural crisis of an industrial basin.

• Possible learning control system in relation to the learning environment

In all three cases reported above, the main learning assessment tool is self-evaluation. In the particular case of Learning Points connected with the ISVOR FIAT experience, as well as self-evaluation there is also evaluation by others that in some cases could, by the student’s request, become a certification of learning results by overtaking the evaluation by others in reference to quali-quantitative specifics of the result produced.

• Reaction of participants to ICT technologies and to the learning control and evaluation system used.

In the three cases cited, ICT even though creating initial concern, did not create particular difficulties for the users.

The same technologies together with management software made up of both the diffused functionalities of the Content Learning Management System (CLMS) platform or integrated by specialist support software for the approach by processes and competences adopted in the Leaning Point case of the ISVOR Fiat experience, supported the tutors either in learning control, or in creating a complex evaluation report.

• Main experiences and references

In this case the experiences are already fully described in the previous points. We attach some of the learning environments layouts and documental elements of experiences presented.

Third case

Videoconference settings that can in turn refer to and use virtual classes/rooms; or can refer to class-laboratory type environments. This second case is represented by “business television”.

• Description of the conditions and technologies that characterise the learning scene

The point to point and multipoint videoconference experience that we present in this document refer in good measure to two situations already described in the second case or rather:

o To the TRIO Centres project

o To the ISVOR Fiat Learning Points

We will report a third case, interesting both for the number of students involved and for the learning efficiency established again by ISVOR Fiat’s experiences in the use of “Business television” to train FIAT Auto sales and post sales staff.

Finally we would like to mention the collaborative working platform case.

• Didactic methodologies predominantly used

“TRIO and Learning Point” cases

In the case of TRIO Centres but also in the ISVOR Learning Point case, one of the classic applications of the videoconference is that of connecting a physical learning environment with experts and/or remote locations thus allowing the development of testimonies or real direct lessons.

The experience developed by ISVOR Fiat can be interesting for distance learning for university courses. For example, we would like to mention “University Diploma in Production Logistics Engineering” course undertaken with two groups in distance learning that were about 800Km apart (Turin and Melfi).

In the latter case, the didactic methodology was traditional with a classroom with the teachers at the head office in Turin and the assistants/tutors at the head office in Melfi. The two classrooms were connected and dialogue was transmitted either via television and videoconferencing or through the internet and intranet. The complete sharing of all the presentations and the group work was assured by an electronic blackboard connected with the local and remote clients’ workstations where the teacher also managed the mixed group work between the classes (see working format attached)

Comments on the methodology and on the virtual classroom situation show absolute respect from the teachers and the participants.

The teachers claimed that technology did not create any obstacle; in fact they ensured that the teacher prepared and carefully planned the classroom situations to make it communicatively efficient and advantageous.

The participants stated that by using the technologies they had the impression of participating in a real class rather than a virtual one; that the memorisation and downloading system both of demonstrations carried out on the electronic blackboard, and of the powerpoint presentations organised by the teacher allowing them to spend less time taking notes was advantageous in the interaction with the teacher.

“Business television” Case

In the “Business television” case, the experience of ISVOR Fiat, the didactic methodology consisted of reproducing in a television studio a series of car sales and/or repair simulations.

The communicative system between the central studio and the 850 sites in Italy and in other European countries was assured beyond the television system, by connecting to the Internet and telephone. In this way ISVOR succeeded to manage a program of about 50 events per year with a media participation of over 2,500 people at each event.

“Collaborative working platform” Case

In the case of collaborative working platforms used for training purposes, quite diffused experiences have been developed in Italy.

The methodological approach is typical of the traditional classroom transferred in a virtual location but with the advantage of being able to see the teacher, or at least hearing his/her voice.

The functions of the system declared by the software distributors are broad both in terms of interactivity with the teacher, and in sharing the documentation and remote control of the work groups.

However, the use of these solutions is still strongly conditioned by sparse diffusion of the “broad band”, this creates interference and suddenly deterioration of quality with particular reference to the video.

• Possible learning control system in relation to the learning environment

“

TRIO” case

The user identification system and that to control learning is exactly the same as the use of online courses on the CLMS platform.

“Learning Point” case

The user identification system and that to control learning is assigned to the Learning Point tutor that uses the support either of the tracking system mainly from the CLMS platform or of other software and offline tools in an integrated way.

In this case we can say that the evaluation is either self-directed or conducted by others.

“Business television” case

In the Business television case, the identification of users that do not directly participate in the lessons occurs either locally by telephone interaction, or through login and password for interaction via internet/intranet.

“Platform of Collaborative working” Case

In this case user identification is managed by the platform that provides the name of the user that is being questioned in the teacher’s workstation. As regards learning control, except for the direct questions asked by the teacher to the group or to a single user, the typical rules of online training through the CLMS platform are valid.

• Participant’s reactions to ICT technologies and to the control and evaluation learning system used

“TRIO” case

The level of participants’ satisfaction has oscillated over time. When there was decreasing interest from the participants, due to the lack of motivation to continue for the reasons already explained, the project administrators progressively made great efforts to improve the training products and tutorial assistance without however completely adopting a “competence based” approach orientated towards know how.

“Learning Point” case

In general, the level of satisfaction is always higher thanks to the tangibility of the acquired competences. Each fall of interest occurring in each LP is due to the lack of funds that has not allowed the renewal of the training offer, this fact, associated, in the case of the LP of Piossasco, to the lack of methodology on the tutor’s part have worsened the situation.

“Business television” case

The level of interest has always been higher for tangibility of professional learning results that the system induces in the users.

“Platform of Collaborative working” case

The level of interest is discretely good and the variations in terms of learning results depend both on the planning level that the teacher puts into act, and on the capacity to animate the learning group in a virtual class setting.

• Main experiences and references

In this case the experiences have already been extensively mentioned when describing single experiences.

Attached are some slides to document and exemplify the experiences of the Production Logistics Engineering course and the Business television activities undertaken out by ISVOR Fiat.

Summary elements of experiences

As you can see from the presentation of experiences, the critical factors of the users’ learning success can be summarised as follows:

• Constant maintenance of the centrality of the user both in terms of training needs and individual needs;

• Rigour in respect of “competence based” approach; that signifies starting from the analysis of what competences the user really nneds to resolve his/her professional and personal problem;

• Methodological rigour in the planning phase both of learning courses and in the preparation of didactic materials and tools, being attentive of alternating processes between acquisition of competences and application either in a classroom-laboratory context, or in the user’s workplace;

• Integrated and intelligent use of ICT useful to the user;

• Personalisation of learning courses with direct involvement of the user both in the training contract phase and in the management and constant control of learning results;

• Strong orientation to the accomplishment of measurable results as fundamental elements to support the motivation to continue;

• Constant control of the learning atmosphere and environment, not abandoning the user to cold technologies but managing communication and human relations;

• To favour the network collaboration between users, orientating them to create practice communities and professional families.

From the point of view of identifying the user within the use of ICT, in order to control and evaluate the learning, the current situation in Italy has almost the only condition of assigning a login and password (automatic and/or from the administrator’s part of the system), while the control of the results is strongly connected with the tracking systems of the CLMS platform.

Note: Digital recognition of training users

In terms of recognition of training users, we have attached an explanatory document of what is in place at the Polytechnic of Turin.

Note: Main Italian experiences and references in an e-learning context

To be able to have a more general and enlarged framework of the Italian experience in an e-learning context we attach series of publications and monographs, collected and/or edited by primary Research, Innovation and Training Institutes of which:

• Anee – Permanent observatory of Assinform (with reference to the Industrial Association of Lombardy)

• CNIPA – National Centre for computing of Public Administration

• INDIRE – National Institute of Documentation for innovation and educational research

• ISVOR Fiat – Corporate University of the FIAT group