在線工作手冊計算機(jī)輔助評估系統(tǒng)【外文翻譯】

1、<p><b>　　畢業(yè)論文（設(shè)計）</b></p><p><b>　　外文翻譯</b></p><p>　　eWorkbook: a Computer Aided Assessment System</p><p>　　Gennaro Costagliola, Filomena Ferrucci, Vitto

2、rio Fuccella, Rocco Oliveto </p><p>　　Dipartimento di Matematica e Informatica, Università di Salerno </p><p>　　Via Ponte Don Melillo, I-84084 Fisciano (SA) </p><p>　　{gcostagl

3、iola, fferrucci, vfuccella, roliveto}@unisa.it</p><p><b>　　Abstract </b></p><p>　　Computer Aided Assessment (CAA) tools are more and more widely adopted in academic environments mixe

4、d to other assessment means. In this paper, we present a CAA Web application, named eWorkbook, which can be used for evaluating learner’s knowledge by creating (the tutor) and taking (the learner) on-line tests based

5、on multiple choice, multiple response and true/false question types. Its use is suitable within the academic environment in a blended learning approach, by providing tutors with a</p><p>　　1 Introduction &l

6、t;/p><p>　　In blended learning the electronic means are mixed with the traditional didactics, in order to train and to assess the learners. Learning Management Systems (LMS), enhanced with collaborative environ

7、ment support, and Computer Aided Assessment (CAA) tools are more and more widely adopted in the academy. At the University of Salerno some systems and platforms have been tested to support blended learning. Even if some

8、good existing systems with LMS capabilities, like OpenUSS (OpenUSS, 2005), Chef (</p><p>　　A state of the art analysis undertaken at our department, which involved several lecturers and students, allowed us

9、to identify the following important requirements for an effective environment for developing and using assessment tests: </p><p>　　? High reusability of the authored content. </p><p>　　? Didacti

10、cs organized in courses and classes. </p><p>　　? Flexible access control system to the tests. </p><p>　　? Quality tracking for the authored content. </p><p>　　? Rich reporting secti

11、on. </p><p>　　A project for a comprehensive Web-based assessment system, named eWorkbook, was then started. The system can be used for evaluating a learner’s knowledge by creating (the tutor) and taking (the

12、 learner) on-line tests based on multiple choice, multiple response and true/false question types. Even though eWorkbook allows the creation of on-line tests for both assessment and self assessment, it was planned above

13、all for summative purposes. The questions are kept in a hierarchical database, that is,</p><p>　　Some other information about the questions is present in the database, such as: difficulty, quality, langua

14、ge, keywords, number of times the question was selected for a test and expected time for a learner to answer.</p><p>　　The tests are composed of one or more sections. This structure facilitates the selecti

15、on of the questions from the database, but it is still useful for the assessment, where it can be important to establish if one section is more important then another to determine the final grade for the test. There are

16、 two kinds of sections: static and dynamic. The difference between them is in the way they allow question selection.</p><p>　　For both the static and the dynamic sections, a macro area in the question databa

17、se must be specified. For a static section, the questions are chosen directly from the sub-tree located by the specified macro area. For a dynamic section, some selection parameters must be further specified, leaving the

18、 system to choose the questions randomly across the sub-tree located by the specified macro area whenever a learner takes a test.</p><p>　　Didactics are organized into courses and classes: the tutors respons

19、ible for a course, manage its class and choose the tests that must be taken by the learners of that class. There are two different lists of tests within the course interface: the valuable and the self assessment test lis

20、ts. Each test in the former list is used to determine the learner’s evaluation, while the latter list is just a guide for the learner to self train and assess. Prerequisites and a maximum number of attempts can b</p&g

21、t;<p>　　Different assessment strategies can be bound to a test, when it is selected for the insertion in the valuable or self-assessment list of a course. The choice of an assessment strategy affects the way in wh

22、ich some parameters concur to determine the grade of the test. The parameters are the following: the weight of a question in the test, the number of distracters’ for a question (only for multiple choice and true/false),

23、the weight of the distracters’ (only for multiple response), bonus and pena</p><p>　　A complete history of learners’ performance on tests of the valuable list is available to the tutor and to the learners th

24、emselves. Each record in the history contains the date and the time when the learner has joined a test, the amount of time needed to finish the test and some information about assessment (test score and state). The detai

25、l of the answers to each question can be seen as well and can be viewed in a printer-friendly format.</p><p>　　The rest of the paper is organized as follows. In Section 2 the main features of the systems are

26、 described in detail. Section 3 is devoted to outlining the architecture of eWorkbook. An example of system use can be found in Section 4. In Section 5, a comparison is made with some interesting systems related to ours.

27、 Some final remarks and a description of future work conclude the paper. </p><p>　　2 The Main Features of eWorkbook </p><p>　　In the following subsections we will outline the main characteristic

28、s of the eWorkbook system. It is worth noting that eWorkbook was intended to be used by a large number of users, so it has a typical LMS didactics organization, based on courses and classes. A course is a place in which

29、the tutors can publish tests and the learners can take them. Learners can only view the tests published in the courses in which they are members. The tutor manages the class and can accept or deny learners’ affil</p&g

30、t;<p>　　2.1 Question Management </p><p>　　An important matter for CAA, and more generally for e-learning, in order to accelerate the teaching and the assessment processes, is the reusability of the au

31、thored content. The on-line material needs a huge initial effort to be created, while it can be easily modified and reused later on. Therefore it is very important that existing material can be easily found, modified and

32、 selected by a tutor who wants to use it for a lesson or a test. There are two main ways to boost the reuse of learning ma</p><p>　　1. Good organization of material kept in an e-learning platform or CAA syst

33、em. </p><p>　　2. Interoperability among systems and platforms, to share and exchange material.</p><p>　　Our system was designed to have a well organized question database to facilitate the tutor

34、 in the question management, share and reuse: the question database of eWorkbook has a hierarchical structure, similar to the directory tree of an operating system. Each item in our database is a disciplinary macro area

35、(internal node) or a question (leaf). The membership of a question to a given macro area is determined by its subject: each macro area is a container of questions that holds items dealing wi</p><p>　　An effo

36、rt for the interoperability has been made supporting the IMS Question & Test Interoperability specification (IMS QTI, 2005): our system can import and export information regarding questions and tests through this wid

37、ely known and adopted XML-based format.</p><p>　　2.1.1 Permissions </p><p>　　Author’s right protection is an important matter too. An e-learning system should offer the tutor the choice to share

38、 his own material or not. In eWorkbook, the owner (the tutor who authored the question) and a permission set are associated to each item. The owner establishes the values for each field of the permission set. A permissio

39、n is a Boolean value that indicates whether other users beyond the owner can perform the action associated to that permission. </p><p>　　For a macro area, the value for the following permissions must be set:

40、 </p><p>　　? Read Permission: the permission to read the property and the contents of this macro area. </p><p>　　? Write Permission: the permission to overwrite the property and manage this macr

41、o area (add </p><p>　　a sub-item to it, delete it). </p><p>　　? Use Permission: the permission to select a question from this macro area for a test. </p><p>　　For a question, the pe

42、rmissions are the following: </p><p>　　? Read Permission: the permission to read the question. </p><p>　　? Write Permission: the permission to delete and overwrite the question. </p><

43、p>　　? Use Permission: the permission to select this question for a test presentation. Its default value is the value of Use Permission of the macro area which this question belongs to. </p><p>　　It’s wort

44、h noting that permissions are a good way to protect author’s right and to avoid that the material owned by a tutor is modified or used without his/her consensus. Other systems only give the possibility to share or not al

45、l of the tutor’s questions. A permission based system gives more flexibility to the system, allowing different grades of item sharing. </p><p>　　2.1.2 Question Metadata </p><p>　　Each question

46、in the database has a metadata set associated to it. Some of the parameters are decided by the tutor when he/she instantiate the metadata and they can be updated later, others are inferred by the system during its use. I

47、nferred metadata are updated whenever a learner submits a test. Metadata are used in question selection in a way that will be clear in the sequel. The following is a list of the metadata fields: </p><p>　　?

48、Language: the human language in which the question is expressed.</p><p>　　? Keywords: a set of keywords that describe the content of the question. </p><p>　　? Use: the aims the question is for.

49、It can be self-assessment, valuable or both. </p><p>　　? Test Occurrence: an inferred field, that is increased by one whenever this question is scheduled for a test . </p><p>　　? Average Answer

50、Time: an inferred field. It can be used on our system because it is able to track the time spent by the learner on each question. </p><p>　　? Difficulty: this field has both an inferred and a tutor chosen va

51、lue. It’s a value between 0 and 1 that expresses a measure of the difficulty of the question, intended as the proportion of learners who get the question correct. The tutor can guess this value at the question creation t

52、ime and can update it during the question’s lifecycle. The system calculates the inferred value with a simple formula. </p><p>　　? Quality: this field is an inferred one. Its value is a measure of how well t

53、his question discriminates between learners. A good question should give full mark to good learners and penalize bad ones. Starting from this information, a great deal of criteria can be adopted. A solution is proposed i

54、n (Lira et al., 1990): it identifies a good question as the one which the better 20% of learners answers well and the worse 20% of learners answers incorrectly. We adopted a common solution applied in I</p><p&

55、gt;　　where the following rules are valid: </p><p>　　o -1 ≤ r ≤ 1, </p><p>　　o x is the series of the results got on the question, </p><p>　　o y is the series of results got on the w

56、hole test.</p><p><b>　　譯文：</b></p><p>　　在線工作手冊：計算機(jī)輔助評估系統(tǒng)</p><p>　　Gennaro Costagliola， Filomena Ferrucci， Vittorio Fuccella， Rocco Oliveto</p><p>　　Dipartim

57、ento di Matematica e Informatica， Università di Salerno</p><p>　　Via Ponte Don Melillo， I-84084 Fisciano (SA)</p><p>　　{gcostagliola， fferrucci， vfuccella， roliveto}@unisa.it</p><

58、;p><b>　　摘要</b></p><p>　　在學(xué)術(shù)環(huán)境中，計算機(jī)輔助評價工具(CAA) 混合其它評價方法越來越多地得到采用。在本文中，我們提出了一個CAA應(yīng)用程序，叫做在線工作手冊，由導(dǎo)師創(chuàng)建，可用于評估學(xué)習(xí)者的知識，以及可以對（學(xué)習(xí)者）進(jìn)行基于多項選擇，多項回答和真或假問題類型的在線測試。它提供額外教師評價工具和遠(yuǎn)距離學(xué)生自我評價系統(tǒng)，它適用于使用混合教學(xué)方法的學(xué)術(shù)環(huán)境中。本

59、文中講述的該工具的主要特點是提供了一個將理論拋諸腦后的系統(tǒng)的設(shè)計大綱。</p><p><b>　　1．簡介</b></p><p>　　為了達(dá)到培訓(xùn)和評估學(xué)習(xí)者的目標(biāo)，常常采用混合式學(xué)習(xí)的電子手段和傳統(tǒng)教學(xué)方法相結(jié)合的方法。學(xué)習(xí)管理系統(tǒng)（LMS）和計算機(jī)輔助評價（CAA）的工具越來越廣泛地在各個學(xué)校里被采用。在Salerno的一所大學(xué)里，對一些平臺和系統(tǒng)進(jìn)行了測試，

60、以支持混合式學(xué)習(xí)。即使現(xiàn)在一些好的系統(tǒng)，就像OpenUSS (OpenUSS， 2005)， Chef (Chef， 2005)， and Sakai (Sakai， 2005)，沒有被測試的工具無法完全滿足我們所有的需要：我們需要一個先進(jìn)的評估工具，它可以幫助講師快速的評估繁重的學(xué)習(xí)任務(wù)，而且可以非常簡單的與LMS系統(tǒng)結(jié)合使用，這點已經(jīng)在我們部門開始實施了。在我們的一個部門對一些教師和學(xué)生進(jìn)行常規(guī)分析的時候，我們確定了在下一個在有

61、效的環(huán)境中的重要需求和需要評估測試內(nèi)容有：</p><p>　　? 高度可重用性的授權(quán)內(nèi)容。</p><p>　　? 舉辦的課程和教學(xué)法的課程。</p><p>　　? 靈活的訪問控制系統(tǒng)的測試。</p><p>　　? 對原創(chuàng)的質(zhì)量跟蹤。</p><p><b>　　? 具體的報告。</b&

62、gt;</p><p>　　當(dāng)一種全面的基于Web的，叫做在線工作手冊的評估系統(tǒng)項目開始時。該系統(tǒng)的評估首先通過（導(dǎo)師）創(chuàng)建，然后學(xué)生（學(xué)習(xí)者）根據(jù)自己所學(xué)的知識對網(wǎng)上的多個選擇題，簡答題，和判斷題進(jìn)行測試。在線工作手冊允許線上創(chuàng)建測試，進(jìn)行評估和自我評估，它的計劃是包括以上所有目的。評估的問題都放在一個層次數(shù)據(jù)庫里，也就是說，他是樹狀結(jié)構(gòu)的，如同在一個操作系統(tǒng)的相同的文件夾里的。在這種結(jié)構(gòu)中，文件可以被看做是問

63、題，以及不論目錄是否為巨大的領(lǐng)域，這通常作為存儲同一主題的問題容器。一個巨大的區(qū)域也可能包含其他宏觀領(lǐng)域。導(dǎo)師可以按自己的意愿自由組織該樹狀結(jié)構(gòu)，例如保存在宏觀領(lǐng)域里的同樣的課程可以根據(jù)這些課程所包括的章節(jié)來分開放置。每一個項目（一個宏觀領(lǐng)域或者一個課題）都有一個所有者，即他們的創(chuàng)作導(dǎo)師。導(dǎo)師可以選擇是否要分享他們的項目或不給予每個項目相關(guān)的權(quán)限。權(quán)限包括讀，寫和使用這些項目。存在數(shù)據(jù)庫中的一些問題有關(guān)的其他信息，如：難度，質(zhì)量，語言，

64、關(guān)鍵字，次數(shù)等問題，它們被挑選出來作為一個測試，然后估計一個評估時間讓學(xué)習(xí)者回答。</p><p>　　這些測試是由一個或多個部分組成的，這種結(jié)構(gòu)有利于從數(shù)據(jù)可以中選擇問題，但是它仍然是用于評估，它可能非常重要，以確定另一個部分是否是更要的一節(jié)，然后再確定測試的最終結(jié)果?，F(xiàn)在有兩部分：靜態(tài)和動態(tài)的。它們之間的區(qū)別是他們的問題的選擇方式。對于這兩種靜態(tài)和動態(tài)部分，在macroarea數(shù)據(jù)庫中是已經(jīng)確定的。對于靜態(tài)部

65、分，問題的選擇是直接從指定的宏觀區(qū)域的子樹里指定的。對于動態(tài)部分，一些動態(tài)參數(shù)必須先確定，然后當(dāng)學(xué)習(xí)者需要被提問時，用隨機(jī)的方法指定宏觀區(qū)域中的子樹所選擇的問題。</p><p>　　教學(xué)的組成包括課程和班級：導(dǎo)師負(fù)責(zé)一個課程，管理其班級和為該班級的學(xué)生選擇測試。在測試過程中有兩種不同的界面：價值觀和自我評估測試列表，每個測試列表前面部分用于確定學(xué)習(xí)者的評價，后面部分僅僅是一個對于學(xué)習(xí)者自我培訓(xùn)和評估的指南列表。

66、值得一提的是，對評估預(yù)先的準(zhǔn)備和最大次數(shù)的嘗試僅僅適用于價值觀測試列表。</p><p>　　不同的評估策略，當(dāng)它被挑選出來插入到有價值的或自我評估列表的項目里，它就可以綁定到一個測試中。一個評估策略的選擇受其中一些等級參數(shù)的影響。參數(shù)如下：試題中的題目數(shù)量，干擾問題的數(shù)量和難度（僅適用于多種選擇和真假選擇），獎勵和懲罰因素。一個評估策略是一個外形，像是一個值的參數(shù)賦值。某些配置是預(yù)先在系統(tǒng)中的，他們被稱為預(yù)定義

67、的評估策略。其他配置可以由導(dǎo)師定義并保存在導(dǎo)師的儲存區(qū)。我們將用他們?yōu)榭蛻舳ㄖ圃u估策略。一個完整的學(xué)習(xí)者的性能測試結(jié)果對其導(dǎo)師和自身非常有用的。當(dāng)學(xué)習(xí)者已經(jīng)參加測試，就會得出總計用了多少時間完成考試，還有關(guān)于評估（考試成績和狀態(tài)）的信息，每個歷史記錄都會包含上述日期和時間等數(shù)據(jù)。詳細(xì)的回答出每個問題的答案可以被視為良好。本文的其余部分組織如下。在第2部分對系統(tǒng)的主要特點進(jìn)行了詳細(xì)介紹。第3節(jié)是專門概述了在線手冊的架構(gòu)。在第4部分有一個系

68、統(tǒng)使用的例子。在第5部分，對我們用的一些有趣的系統(tǒng)作了一些比較。最后是最終的結(jié)論和對未來工作介紹并以它們來結(jié)束本文。</p><p>　　2．在線工作手冊的主要特點</p><p>　　在下面的小節(jié)中，我們將要闡述在線工作手冊系統(tǒng)的主要特點。在線工作手冊是給大量用戶使用的，因此它的過程具有典型的LMS教學(xué)組織方法。導(dǎo)師可以在其中發(fā)布測試，學(xué)習(xí)者可以在其中得到測試。學(xué)生只能查看自己團(tuán)隊的，已

69、經(jīng)發(fā)布的測試。導(dǎo)師管理自己的團(tuán)隊，比如可以接受或者拒絕學(xué)生的要求和開除團(tuán)隊中的學(xué)生。</p><p><b>　　2.1管理問題</b></p><p>　　為了加快教學(xué)和評估過程，使授權(quán)的內(nèi)容可重用性變高，使得CAA在線學(xué)習(xí)變的更加重要和普遍了。當(dāng)要求線上的資源能夠被輕松修改和重用的時候，就需要重新建立一個系統(tǒng)，這將會付出巨大的成本。因此，使用現(xiàn)有的資源非常重要的，

70、現(xiàn)在的資源很容易得到，然后由導(dǎo)師修改和選擇誰需要一個教程或者測試?，F(xiàn)在主要有兩種途徑來提高學(xué)習(xí)的材料再利用：</p><p>　?。?）保存在一個在線學(xué)習(xí)平臺或CAA系統(tǒng)里的良好組織的材料。</p><p>　?。?）系統(tǒng)和平臺間的互用性，可以共享交流材料。</p><p>　　我們的系統(tǒng)有一個組織良好的試題庫，以便導(dǎo)師進(jìn)行問題管理，共享和重用：在線工作手冊有一個層

71、次結(jié)構(gòu)，類似操作系統(tǒng)的目錄樹。我們的數(shù)據(jù)庫中每個項目都是一個內(nèi)部節(jié)點或者一個葉。一個項目有沒有資格存放在宏觀領(lǐng)域中取決于它的主題：每個宏觀區(qū)域都是一個容器，像保存特殊的主題一樣用于保存一個項目。它可以在宏觀區(qū)域的分領(lǐng)域進(jìn)一步的分離，這樣可以保持問題屬于一個更加特殊的項目。問題的類別允許有多個選擇，多個響應(yīng)和真假。導(dǎo)師可以選擇問題僅僅用于評估活動，或者用于自我評估，或者即用于評估活動又用于自我評估。一種可以互用的成果已經(jīng)取得了成功，它支持

72、IMS Question和共享測試系統(tǒng) (IMS QTI， 2005):我們的系統(tǒng)可以通過眾所周知的基于以XML為基礎(chǔ)的格式的方法導(dǎo)入和導(dǎo)出問題和測試的的相關(guān)信息。</p><p><b>　　2.1.1權(quán)限</b></p><p>　　原創(chuàng)者的權(quán)利保護(hù)是一個重要問題。在線學(xué)習(xí)系統(tǒng)提供導(dǎo)師是否愿意分享自己的資源。在在線工作手冊中，擁有者（即作者）和權(quán)限設(shè)置關(guān)系到每個項

73、目。擁有者對各個價值不同項目區(qū)域設(shè)置不同的權(quán)限。權(quán)限是一個布爾值，允許是否被其他用戶使用該資源。對于宏觀區(qū)域，以下權(quán)限必須設(shè)置：</p><p>　　?閱讀權(quán)限：讀取本宏觀區(qū)域性能和內(nèi)容的權(quán)限。</p><p>　　?修改權(quán)限：修改和管理這個宏觀區(qū)域（如添加一個分項目或刪除項目）的權(quán)限。</p><p>　　?使用權(quán)限：從宏觀區(qū)域里選擇一個用于測試問題的權(quán)限。<

74、;/p><p>　　對于一個問題，權(quán)限如下：</p><p>　　?閱讀權(quán)限：讀取問題的權(quán)限。</p><p>　　?修改權(quán)限：權(quán)限刪除和覆蓋的問題。</p><p>　　?使用權(quán)限：選擇一個問題進(jìn)行測試。它的默認(rèn)值是macroarea使用許可的情況下，該問題屬于的權(quán)限。值得一提的是，權(quán)限是一個很好的來保護(hù)作者的權(quán)利的方法，可以避免作者擁有的資源

75、被修改或者沒經(jīng)過作者的允許時被使用。其他系統(tǒng)只會給出可能共享或者不屬于導(dǎo)師的所有問題。因此，可以看出一個權(quán)限系統(tǒng)提供了更多的靈活的方法到系統(tǒng)中，允許項目分權(quán)限等級被使用。</p><p>　　2.1.2元數(shù)據(jù)的問題</p><p>　　數(shù)據(jù)庫中的每個問題都有一個與之相關(guān)的元數(shù)據(jù)集。有些參數(shù)是由導(dǎo)師決定的，當(dāng)他或她實例化元數(shù)據(jù)時，它們就得以被更新，其他信息都是在系統(tǒng)使用期間更新的。無論何時，

76、當(dāng)一個學(xué)生提交了一個測試，元數(shù)據(jù)就會更新。元數(shù)據(jù)是是用于問題的選擇的，這將在接下來說明。以下是列表中的元數(shù)據(jù)字段：</p><p>　　?語言：即是人類語言。</p><p>　　?關(guān)鍵字：關(guān)鍵字是用來描述問題的內(nèi)容概況的。 </p><p>　　?用途：設(shè)置問題的目的。它可以自我評估，評估價值或兩者兼有之。</p><p>　　?測試事件：

77、為了測試而作出的對問題的安排。</p><p>　　?平均應(yīng)答時間：它可以用在我們的系統(tǒng)上，因為它能夠記錄每一個問題上花費(fèi)的學(xué)習(xí)時間。</p><p>　　?難度：這部分既有預(yù)測的，也有導(dǎo)師選擇的值。這是一個介于0和1之間的價值，這個值表示問題困難度的大小，這個值受學(xué)習(xí)者答題時的正確率影響。導(dǎo)師可以在創(chuàng)建這個問題的時候猜測這個值的大小，而且可以在這個問題的生命周期期間更新它。最后該系統(tǒng)用簡

78、單的公式得出這個準(zhǔn)確的值。</p><p>　　?質(zhì)量：質(zhì)量是描述一個問題能否正確地評估學(xué)生。一個好的問題，應(yīng)該可以區(qū)分出好的學(xué)生和差的學(xué)生。因此，有很多標(biāo)準(zhǔn)可以被采納。一個解決方案被提出來了（Lira et al.， 1990）：它定義了一個好的問題應(yīng)該是20%學(xué)生回答的很好，20%的學(xué)生回答錯誤。我們采取一種常見的方案用于解決項目分析，該計算質(zhì)量的方案就是皮爾森提出的解決相關(guān)性問題和為達(dá)到測試目的的一種方案。