外文翻譯--計(jì)算機(jī)輔助評(píng)估來(lái)測(cè)試高等水平的學(xué)習(xí)成果（節(jié)選）

1、<p>　　2100單詞，1.2萬(wàn)英文字符，3600漢字</p><p>　　出處：Duke-Williams E, King T. Using computer-aided assessment to test higher level learning outcomes[J]. © loughborough university, 2001.</p><p><

2、;b>　　外文原文</b></p><p>　　USING COMPUTER AIDED ASSESSMENT TO TEST HIGHER LEVEL LEARNING OUTCOMES</p><p>　　Terry King and</p><p>　　Emma Duke-Williams</p><p>　　Usin

3、g Computer-Aided Assessment to Test Higher</p><p>　　Level Learning Outcomes</p><p>　　Terry King and Emma Duke-Williams</p><p>　　Department of Information Systems</p><p>

4、;　　University of Portsmouth</p><p>　　Buckingham Building</p><p>　　Portsmouth PO1 3HE</p><p>　　Email: terry.king@port.ac.uk</p><p>　　Tel: +44 (0) 23 9284 6426</p>

5、<p><b>　　Abstract</b></p><p>　　This paper sets out an approach using a revised Bloom's taxonomy of learning objectives for the careful design of objective questions to assist in the ass

6、essment of higher learning outcomes (HLO’s) and details the creation and evaluation of a variety of such questions. This has been done within the context of two constraints; the use of popular, commercially available com

7、puter-aided assessment (CAA) software, specifically Question Mark Perception and Half-Baked Hot Potatoes, and the assumption o</p><p><b>　　Keywords</b></p><p>　　Computer-aided Assess

8、ment, Bloom’s Taxonomy, Learning Objectives, Objective Testing, Computer-based Testing</p><p>　　Background</p><p>　　Students in higher education engage in a variety of learning activities with t

9、he aim of attaining certain defined learning outcomes. As students progress to Level 3 and post-graduate work, it is accepted that they engage increasingly in activities designed to develop skills and abilities which are

10、 considered to be of a higher cognitive complexity (Zakrzewski and Steven, 2001). Holzl (2000) places emphasis on the development of certain graduate attributes in the cognitive domain such as critical </p><p&

11、gt;　　This paper sets out an approach using a revised Bloom's taxonomy of learning objectives for the careful design of objective questions to assist in the assessment of higher learning outcomes, and details the crea

12、tion and evaluation of a variety of such questions. This has been done within the context of two constraints; the use of popular, commercially available CAA software and the limited availability of learning technician/te

13、chnologist support. It assumes an environment in which the authors an</p><p>　　Classification of Learning Outcomes</p><p>　　Bloom's taxonomy of learning objectives has been chosen as the fra

14、mework for approaching the problem of assessing for higher learning outcomes (Bloom et al,1956). The reason for this follows Delgano (1998) who considered other schemes for classifying learning outcomes but came down in

15、favour of Bloom's taxonomy because it had sufficiently detailed categories to allow outcomes to be mapped clearly onto learning activities, and was in widespread use so designers did not have to learn an additional&l

16、t;/p><p>　　Bloom's taxonomy of learning objectives (Bloom et al, 1956) attempted to classify forms of learning into three categories: cognitive, affective and psychomotor domains. Within the cognitive domai

17、n, Bloom identified six levels of learning which represented increasing levels of cognitive complexity from the lowest level of Knowledge (or remembering) through Comprehension, Application, Analysis,Synthesis and Evalua

18、tion. Each level encompassed those below it, so, for example,analysis could only occu</p><p>　　Objective Question Design for Higher Learning Outcomes</p><p>　　Advice is available for the design

19、of effective objective tests by providing general guidelines for question design, and employing grids and matrices to plot content against learning levels and outcomes (Heard et al,1997; Rolls and Watts, 1998).However mu

20、ch less direction is given on how to design the assessment questions themselves. Because of this, the design of objective questions to test higher learning outcomes (HLO's) often follows one of three approaches:</

21、p><p>　　Derivation from the verbs associated with HLO's. This can misleading. For example,any question with the verb 'judge' is assumed to be an Evaluation level question, but in fact students are a

22、sked to judge on a variety of criteria, and, if the criteria is understanding of a theory, then the question will be at a Comprehension level. Often such criteria are not made explicit to students so that they are left c

23、oping with ambiguity</p><p>　　Extrapolation from existing subjective examination questions. This can result in objective questions which are unclear as to which learning level they apply. This is hardly surp

24、rising as examination questions are generally not subjected to the rigorous examination applied to objective questions and exactly what they assess can be open to debate. Use of exemplars. For example, those described by

25、 Mackenzie (1999), Carneson et al. (no date), and Heard et al.(1997) are effective, but are limited bec</p><p>　　None of these are particularly successful. What is needed is a systematic framework for positi

26、oning questions for particular learning outcomes.</p><p>　　A framework which offers many possibilities is the revision to Bloom’s taxonomy by Anderson and Krathwohl (2001) and co-workers which results in the

27、 basic table shown in Figure 1. The six levels remain but each has been replaced by its matching verb – Remember, Understand, Apply, Analyse, Evaluate and Create – in order to facilitate the writing of learning objective

28、s. Create is now the last and highest level of learning as they consider that evaluation is a necessary step which precedes any gene</p><p>　　The three higher learning levels which were initially of main int

29、erest for constructing CAA questions were subdivided in detail:</p><p>　　Analyse - encompassing differentiating or distinguishing, organising or structuring, and deconstructing (which concerns determining th

30、e values underlying presented material).</p><p>　　Evaluate - which breaks down into the two processes of checking for internal consistency, and critiquing which involves judging against external criteria.<

31、;/p><p>　　Create - which involves generative processes such as hypothesizing, planning,designing, and producing or constructing.</p><p>　　Figure 1: Modified basic table of Bloom's learning obje

32、ctives (Anderson and Krathwohl,2001) showing the distribution of objective questions with HLO's used in the paper.</p><p>　　The table in Figure 1 has been extended by the addition of a knowledge dimensio

33、nto help 'educators distinguish more closely what they teach' and by implication what they are assessing. This dimension is detailed in the columns on the left of Figure 1,with the different forms of knowledge co

34、vering:</p><p>　　Factual Knowledge. The basic details of the content of a course which students must know to make sense of the discipline, divided into knowledge of terminology or specific details and elemen

35、ts. Itemised knowledge before interrelationships are considered here.</p><p>　　Conceptual Knowledge. Encompassing the knowledge of classifications and categories, principles and generalisations, theories, mo

36、dels and structures.Procedural Knowledge. This considers the knowledge of subject-specific skills,algorithms, techniques and methods, and also the knowledge of the criteria used in determining when to use specific proced

37、ures. It is the knowledge of 'how to do something'.</p><p>　　Metacognitive Knowledge. That class of knowledge by which students know how they come to know and learn. It includes the conscious applica

38、tion of cognitive strategies by students and their own self-knowledge of learning strengths and styles.</p><p>　　In using this framework for the construction of objective questions to test HLO's,certain

39、areas of the table in Figure 1 have been excluded, viz. meta cognitive knowledge which is not subject-specific, and the two lowest learning levels of learning which are not applicable to this investigation. </p>&

40、lt;p>　　Also it was the original intention to include the highest level of learning 'Create' within the study. Anderson and Krathwohl (2001) are quite specific in the activities which are included within this l

41、evel of learning, viz. generating, planning and producing. Each of the many results of such activities ie. alternative hypotheses, research plans, designed procedures, an invention or a construction, are by their very na

42、ture both unique and equally valid. There must be many 'correct' answers, whic</p><p>　　Figure 2: The student is required to drag symbols into specified locations to 'construct' a working dia

43、gram for a given problem.</p><p>　　Figure 3: The student is asked to correct an incorrect layout according to specific criteria using drag-and-drop.</p><p>　　However further consideration of the

44、se questions against the framework lead to the conclusion that they were not at the highest learning level of create but essentially an analysis of material with a view to determining how elements are organised or (re)st

45、ructured. In view of this it was decided to omit further consideration of the create learning outcome and to aim questions largely at the levels of analyse and evaluate, with some more exploration of apply level question

46、s. The latter have the s</p><p>　　22 objective questions for assessing HLO's were constructed for implementation in two commercially available CAA software packages, Question Mark Perception and Half-Bak

47、ed Hot Potatoes. The questions were devised for formative assessment for two course units on the MSc in Information Systems, one for basic multimedia theory, and the other for educational theory underpinning the developm

48、ent of computer-aided learning. The questions fell into three categories:</p><p>　　A. 13 questions devised solely using the revised Bloom's framework.</p><p>　　B. 6 questions devised by usin

49、g exemplars.</p><p>　　C. 3 questions adapted directly from past exam papers in multimedia theory.</p><p><b>　　譯文：</b></p><p>　　計(jì)算機(jī)輔助評(píng)估來(lái)測(cè)試高等水平的學(xué)習(xí)成果</p><p>　　

50、特里金和艾瑪杜克-威廉姆斯</p><p>　　用電腦輔助評(píng)估來(lái)測(cè)試高層次的學(xué)習(xí)成果</p><p>　　特里金和公爵艾瑪-威廉姆斯</p><p>　　樸次茅斯大學(xué)信息系統(tǒng)系</p><p>　　電子郵件：terry.king @ port.ac.uk</p><p>　　電話：+44（0）23 9284 6426&

51、lt;/p><p><b>　　摘要</b></p><p>　　本文陳述了一種改進(jìn)的Bloom學(xué)習(xí)目標(biāo)分類法，即對(duì)客觀問(wèn)題進(jìn)行精心設(shè)計(jì)去輔助評(píng)估高等教育成果，并詳細(xì)說(shuō)明了對(duì)各種此類問(wèn)題的創(chuàng)新和評(píng)估。該方法受到普遍使用的商用輔助評(píng)估軟件的制約和受知覺(jué)、學(xué)習(xí)和技術(shù)專家的支持。它研究的問(wèn)題是HLO客觀題所固有的設(shè)計(jì)（專門(mén)的應(yīng)用，分析和評(píng)價(jià)的水平），首先它介紹了該系統(tǒng)的設(shè)計(jì)，然

52、后對(duì)該系統(tǒng)做出整的框架。它所研究得是建設(shè)模式和兩個(gè)研究生的信息系統(tǒng)課程項(xiàng)目，主要涉及的是兩批學(xué)生，共37個(gè)，和形成性評(píng)價(jià)設(shè)計(jì)等22個(gè)客觀題的評(píng)價(jià)。該方法首先是在民航局進(jìn)行試運(yùn)行，然后再反饋。這個(gè)過(guò)程中在評(píng)分和交付方面出現(xiàn)了軟件中關(guān)鍵部分的分歧。本文主要考察的是統(tǒng)計(jì)指標(biāo)質(zhì)量問(wèn)題（設(shè)施和歧視指數(shù)），和通過(guò)學(xué)生的訪談從而得出最好的問(wèn)題類型以及學(xué)生準(zhǔn)備測(cè)試、備考要注意的關(guān)鍵問(wèn)題。本文總結(jié)了并與考試的主要優(yōu)點(diǎn)以及在使用HLO的民航局的資源這一過(guò)程

53、中存在的缺點(diǎn)。</p><p><b>　　關(guān)鍵詞</b></p><p>　　電腦輔助評(píng)估，Bloom的分類，學(xué)習(xí)目標(biāo)，目標(biāo)測(cè)試，計(jì)算機(jī)輔助測(cè)試</p><p><b>　　背景</b></p><p>　　受高等教育的學(xué)生都致力于各種學(xué)習(xí)，從而獲得一些特定的學(xué)習(xí)成果。當(dāng)學(xué)生進(jìn)步到第3等級(jí)或研究生

54、的工作時(shí)，他們從事于設(shè)計(jì)，開(kāi)發(fā)技能和提高認(rèn)知復(fù)雜性（Zakrzewski和Steven，2001年）能力等活動(dòng)。 Holzl（2000年）強(qiáng)調(diào)注重畢業(yè)生的發(fā)展，尤其是在批判性思維領(lǐng)域和知情判斷的認(rèn)知領(lǐng)域，以及深入的知識(shí)領(lǐng)域中所涉及的信息管理組織和分析的能力，跨學(xué)科的觀點(diǎn)和解決問(wèn)題要求的評(píng)估和創(chuàng)造。然而這些屬性的發(fā)展需要適當(dāng)?shù)膶W(xué)習(xí)活動(dòng)和目的的評(píng)估。因此，高等教育體系中出現(xiàn)了某些需要的評(píng)估體系。在評(píng)估中，首先要促進(jìn)以上幾種能力的發(fā)展從而形成

55、自我評(píng)估，以及相關(guān)的評(píng)估，然后要尋找一種在相關(guān)知識(shí)領(lǐng)域中實(shí)踐的估計(jì)方法。通過(guò)參與這些活動(dòng)，接受適當(dāng)?shù)姆答?，學(xué)生不僅可以學(xué)習(xí)到知識(shí)還可以取得一定的進(jìn)步。傳統(tǒng)的反饋是導(dǎo)師和學(xué)生面對(duì)面或?qū)W生組員之間的反饋，但是由于學(xué)生人數(shù)的增加和教育資源配置日益稀少，使得這種方式的反饋越來(lái)越少。在總結(jié)性評(píng)估中，要進(jìn)一步的對(duì)學(xué)生專業(yè)素養(yǎng)的評(píng)估能力進(jìn)行測(cè)評(píng)。傳統(tǒng)的基于紙張的書(shū)面考試涉及到標(biāo)記，主觀性，偏見(jiàn)等因素的影響，因此，題型重點(diǎn)放在客觀題上，從而查看這種評(píng)估

56、模式是否適用于復(fù)雜的認(rèn)知能力相</p><p>　　本文陳述了一種改進(jìn)的Bloom學(xué)習(xí)目標(biāo)分類法即對(duì)客觀問(wèn)題進(jìn)行精心設(shè)計(jì)去輔助評(píng)估高等教育成果并詳細(xì)說(shuō)明了對(duì)各種此類問(wèn)題的創(chuàng)新和評(píng)估。該方法受商用電腦輔助評(píng)估軟件的普遍使用的制約和問(wèn)題標(biāo)記 知覺(jué)、學(xué)習(xí)和技術(shù)專家假設(shè)的支持。它假定一個(gè)環(huán)境，然后在這個(gè)環(huán)境中使作者和許多其他講師發(fā)現(xiàn)自己，在這過(guò)程中，他們需要全心全意地投入問(wèn)題的創(chuàng)造過(guò)程，熟練地應(yīng)用軟件應(yīng)用程序，但是他們不

57、需要任何的編程技能。</p><p><b>　　分類學(xué)習(xí)成果</b></p><p>　　Bloom的學(xué)習(xí)目標(biāo)分類已被選定作為評(píng)估較高學(xué)習(xí)成果（Bloom等人，1956年）的框架。其原因如下：Delgano（1998）雖研究的是其他分類學(xué)習(xí)結(jié)果的計(jì)劃，但卻完全贊同了Bloom的目標(biāo)分類。因?yàn)锽loom的分類方法有足夠詳細(xì)的分類，能讓結(jié)果清楚地被映射到所對(duì)應(yīng)的學(xué)習(xí)活動(dòng)

58、，Bloom的分類法的廣泛使用使得設(shè)計(jì)者不再需要額外的學(xué)習(xí)計(jì)劃。在樸次茅斯大學(xué)，評(píng)估策略與Bloom分類完全地聯(lián)系在一起，其中后者尤其重要。在制定學(xué)習(xí)目標(biāo)時(shí)，要竭盡全力地熟悉講師和Bloom分類。</p><p>　　Bloom的分類學(xué)習(xí)目標(biāo)（Bloom等人，1956年）試圖把學(xué)習(xí)形式分為三類：認(rèn)知，情感和精神領(lǐng)域。在認(rèn)知領(lǐng)域，Bloom確定了六個(gè)學(xué)習(xí)級(jí)別，從知識(shí)的最低級(jí)別（記?。╅_(kāi)始，通過(guò)理解，應(yīng)用，分析，綜合

59、與評(píng)估逐漸地認(rèn)知復(fù)雜性。每個(gè)級(jí)別都包含它下面的所以級(jí)別，舉例來(lái)說(shuō)，分析只有在具有認(rèn)識(shí)事實(shí)或理解其他知識(shí)的能力時(shí)才可能發(fā)生。最低的三個(gè)級(jí)別被認(rèn)為高等教育水平的'基礎(chǔ)思想'（Ryan和Frangenheim，2000）。每個(gè)級(jí)別相關(guān)聯(lián)的某些學(xué)習(xí)成果表現(xiàn)為'動(dòng)詞'，如回憶，繪制，計(jì)算，分類，設(shè)計(jì)或評(píng)估。更高的學(xué)習(xí)成果將通過(guò)更復(fù)雜的學(xué)習(xí)認(rèn)知程度來(lái)反映。人們常常想當(dāng)然地認(rèn)為提供一個(gè)正確答案的測(cè)試只適用于學(xué)習(xí)的最低水

60、平。雖然這從來(lái)都是錯(cuò)的，但在客觀測(cè)試中，最高的三個(gè)級(jí)別是分析，綜合和評(píng)價(jià)，現(xiàn)在可以考慮更為合適的等級(jí)適用測(cè)試的進(jìn)展。然而這從來(lái)都是錯(cuò)的，隨著民航局的進(jìn)步和發(fā)展，測(cè)試的三個(gè)最高水平-分析、綜合和評(píng)估現(xiàn)在具有一定的適用性。</p><p>　　高等教育學(xué)習(xí)成果客觀問(wèn)題設(shè)計(jì)</p><p>　　使用提供問(wèn)題設(shè)計(jì)的一般準(zhǔn)則和采用網(wǎng)格和矩陣來(lái)提高學(xué)習(xí)水平（1997年的內(nèi)容; Rolls和Watts，

61、1998 ）的建議可用于有效地客觀的測(cè)試設(shè)計(jì)，但是很少根據(jù)評(píng)估問(wèn)題本身而設(shè)計(jì)的。正因?yàn)槿绱?，測(cè)試更高的學(xué)習(xí)成果（HLO的）客觀題的設(shè)計(jì)，經(jīng)常是如下三種方法之一：</p><p>　　從HLO衍生相關(guān)的動(dòng)詞。任何與動(dòng)詞'判斷'有關(guān)的問(wèn)題都假定為評(píng)估水平的問(wèn)題，這可能會(huì)產(chǎn)生誤導(dǎo)，但實(shí)際上學(xué)生們要從各種標(biāo)準(zhǔn)進(jìn)行判斷，如果標(biāo)準(zhǔn)是理論的理解，那么問(wèn)題將會(huì)是在理解級(jí)別上。通常這種標(biāo)準(zhǔn)并未明確，所以學(xué)生只能根據(jù)

62、現(xiàn)有的主觀試題來(lái)進(jìn)行模糊地推斷。</p><p>　　現(xiàn)有的主觀試題的外推法。這可能會(huì)產(chǎn)生不明確的學(xué)習(xí)等級(jí)問(wèn)題。客觀問(wèn)題的試題一般不會(huì)受到嚴(yán)格的考試制約，他們的評(píng)估可以公開(kāi)地辯論。</p><p>　　使用范例。如由麥肯齊（1999），Carneson等人所描述的，聽(tīng)取客觀模式是有效的，但也是有限得，因?yàn)?，?dāng)產(chǎn)生客觀題類型的模板時(shí)，他們不再提供正式的講師和設(shè)計(jì)方法來(lái)創(chuàng)造設(shè)計(jì)新問(wèn)題的格式。&

63、lt;/p><p>　　這些都不是特別成功。我們需要的是一個(gè)特定的學(xué)習(xí)成果的定位問(wèn)題以及系統(tǒng)的框架。</p><p>　　一個(gè)框架，可以提供許多可能性，這是由安德森和Krathwohl（2001年）和同事對(duì)Bloom的分類法的修訂，基本表如圖1所示。六個(gè)級(jí)別雖然存在，但是每個(gè)都被其匹配的動(dòng)詞取代 - 記住，理解，應(yīng)用，分析，評(píng)估和創(chuàng)新 - 為了便于學(xué)習(xí)目標(biāo)的確立?，F(xiàn)在創(chuàng)新是學(xué)習(xí)的最后和最高的級(jí)

64、別，與此同時(shí)他們認(rèn)為評(píng)價(jià)也是一個(gè)必要的步驟，它先于任何生成過(guò)程。</p><p>　　這三個(gè)較高的學(xué)習(xí)等級(jí)最初是以民航局建設(shè)問(wèn)題的主要核心，同時(shí)對(duì)它進(jìn)行了詳細(xì)的劃分：</p><p>　　分析 - 包括鑒別或區(qū)分，組織或結(jié)構(gòu)，與解構(gòu)（即有關(guān)決定提出的價(jià)值觀基礎(chǔ)材料）。評(píng)估-可以分解成兩個(gè)進(jìn)程，一個(gè)是檢查內(nèi)部的一致性，另一個(gè)是判斷外部標(biāo)準(zhǔn)。創(chuàng)建-這一層次中涉及到諸如假設(shè)，規(guī)劃，設(shè)計(jì)和生產(chǎn)或建

65、設(shè)的生成過(guò)程。</p><p>　　圖1：改良的Bloom的學(xué)習(xí)目標(biāo)基本表（Anderson和Krathwohl，2001年），它顯示了本文中HLO的文件中客觀題的分布。</p><p>　　圖一表中知識(shí)層面的延長(zhǎng)使得教育工作者更加緊密地分辨他們教的是什么，并提醒他們正在評(píng)估什么。圖1的左側(cè)通過(guò)涵蓋知識(shí)的不同形式詳細(xì)闡述了這一層面。</p><p>　　事實(shí)性知識(shí)。

66、即課程內(nèi)容的基本資料，學(xué)生必須知道如何劃分術(shù)語(yǔ)或知識(shí)的具體細(xì)節(jié)和要素以及理解他們之間的關(guān)系。</p><p>　　理性認(rèn)識(shí)。涵蓋了分類和類別，原則和概括，理論，模式和結(jié)構(gòu)的知識(shí)。</p><p>　　程序性知識(shí)。這被認(rèn)為個(gè)別學(xué)科的技巧，算法，技術(shù)和方法的知識(shí)，也是在確定何時(shí)使用特定的程序所使用的標(biāo)準(zhǔn)的知識(shí)。它是如何做的知識(shí)。</p><p>　　元認(rèn)知知識(shí)。這類知識(shí)

67、使得學(xué)生知道他們是如何認(rèn)識(shí)和學(xué)習(xí)的。它包括學(xué)生自覺(jué)地運(yùn)用認(rèn)知策略和自我學(xué)習(xí)知識(shí)的優(yōu)勢(shì)和風(fēng)格。</p><p>　　用客觀題構(gòu)建的框架來(lái)測(cè)試HLO時(shí)，圖一表中的某些領(lǐng)域已經(jīng)被排除在外，即未特定的元認(rèn)知知識(shí)和兩個(gè)不適用這項(xiàng)調(diào)查學(xué)習(xí)的最低學(xué)習(xí)水平，但它最初是包括學(xué)習(xí)'創(chuàng)新'這一最高級(jí)別的。</p><p>　　Anderson和Krathwohl（2001年）對(duì)學(xué)習(xí)級(jí)別進(jìn)行了相當(dāng)

68、具體的研究，它包含生成，規(guī)劃和生產(chǎn)活動(dòng)。替代假設(shè)，研究計(jì)劃，設(shè)計(jì)程序，一項(xiàng)發(fā)明或構(gòu)建，每一個(gè)這些活動(dòng)的成果都具有其本身的獨(dú)特性和同等的效力。在客觀測(cè)試中必須有許多'正確的答案，但它涉及這個(gè)外部的客觀測(cè)試的職權(quán)范圍，需要一個(gè)（且只有一個(gè)）完全有效的響應(yīng)級(jí)別，并有可能使民航局內(nèi)能自動(dòng)標(biāo)記。學(xué)生可以拖動(dòng)可選標(biāo)記到模板（圖2）或改善問(wèn)題中不良的子構(gòu)建布局（圖3），這樣進(jìn)行一系列的反復(fù)的探索和嘗試。</p><p>

69、;　　圖2：學(xué)生根據(jù)給定的問(wèn)題，拖動(dòng)符號(hào)到指定的位置，從而構(gòu)建工作圖。</p><p>　　圖3：要求學(xué)生根據(jù)拖和放的具體標(biāo)準(zhǔn)來(lái)改正不正確的布局。</p><p>　　通過(guò)對(duì)框架這一問(wèn)題的進(jìn)一步思考，得出它們不在創(chuàng)造這一學(xué)習(xí)級(jí)別上的結(jié)論以及如何對(duì)元素進(jìn)行重組的分析。</p><p>　　鑒于此，決定省略了創(chuàng)建學(xué)習(xí)成果的進(jìn)一步審議，把目標(biāo)主要放在分析和評(píng)估水平的問(wèn)題上

70、，并在應(yīng)用層次問(wèn)題上進(jìn)行更多的探索。應(yīng)用層次上的探索是熟悉的執(zhí)行任務(wù)的子類別或是應(yīng)用程序上陌生任務(wù)的實(shí)施 。</p><p>　　在Question Mark Perception和 Half-Baked Hot Potatoes這兩個(gè)商用軟件的構(gòu)建執(zhí)行中，應(yīng)用了22道評(píng)估HLO的客觀問(wèn)題。這些客觀問(wèn)題是為這兩個(gè)信息系統(tǒng)項(xiàng)目的形成性評(píng)估而設(shè)計(jì)的，其中一個(gè)項(xiàng)目是多媒體基礎(chǔ)，另一個(gè)是電腦輔助學(xué)習(xí)的發(fā)展理論。而這些問(wèn)題