(PICTURE)(PICTURE) (PICTURE)Europäis hes Patentamt
(1 9) (PICTURE) European Patent Offi_e
(PICTURE) Offi_e euroPéen des brevets (1 1 ) E _ O 664 O4_ B_
(1 2) EUROPEAN PATENT _PECl FICATlON
(45) Date of pUbliCation and mention (51 ) _nt C_.6_. _O9B _JOO, _OgB _/O2,
of the grant of the patent_. _ogB _/o__
O8.12.1999 Bulletin 1999l49
(2 ) (86) International application number_.
1 AppliCatiOn nUmber'. 93923861 _4 p__lu_g3logi6i
(22) Date of filing_. 12.1 O.1993 (8_) _nternat__ona_ pub___Cat__on number..
(PICTURE)WO 94lO9466 (28.O4.1994 Gazette 1994l1 O)
(54) _Y_TEM AND METHOD FOR COMPUTER BA_ED TE_TING
SYSTEM UND VERFAH REN ZUR COMPUTERUNTERSTUETZTEN PRUEFUNG
(PICTURE)(PICTURE)
SYSTEME ET PROCEDE D' EXAMEN IN FORMATISE
__
__
O
_
_ N w. h. . h f h b_. . f h . f h f h E .
_ Ote'_ It In nIne mOnt S rOm t e PU ICatIOn O t e mentIOn O t e 9rant O t e UrOPean Patent_ anY PerSOn maY 9IVe
o nOtiCe tO the EUrOpean Patent OffiCe Of OppOSitiOn tO the EUrOpean patent granted. NOtiCe Of OppOSitiOn Shall be filed in
ii a written reasoned statement. It shall not be deemed to have been filed until the opposition fee has been paid. (Art.
w (PICTURE)99(1 ) European Patent Convention). Printed by Jouve, 75OO1 PARIS (FR)
(PICTURE)
EP O 664 O41 B1
Des_ription
5 [OOO1] For many years standardized tests have been administered to examinees for various reasons such as for
educational testing or for evaluating particular skills. For instance, academic skills tests (e.g. SATs, LSATs, GMATs,
etc.) are typically administered to a large number of students. Results of these tests are used by colleges, universities
and other educational institutions as a factor in determining whether an examinee should be admitted to study at that
educational institution. Other standardized testing is carried out to determine whether or not an individual has attained
1O a specified level of knowledge, or mastery, of a given subject. Such testing is referred to as mastery testing (e.g.
achievement tests offered to students in avariety of subjects and the results being used for college entrance decisions).
[OOO2] Figure 1 depicts a sample question and sample direction which might be given on a standardized test. The
stem 4, the stimulus 5, responses 6 and directions 7 for responding to the stem 4 are collectively referred to as an
item. The stem 4 refers to a test question or statement to which an examinee is to respond, e.g., question 1 3. The
15 stimulus 5 is the text and/or graphical information (e.g., a map, scale, graph, or reading passage) to which a stem may
refer. Often the same stimulus is used with more than one stem. Some items do not have a stimulus. Items having a
common stimulus are defined as a set. In Figure 1 , questions 1 3 and 1 4 refer to stimulus 5 and therefore form a set.
Items sharing common directions are defined as a group. Thus, questions 8-27 form a group. Only questions 8-1 4,
however, are shown in Figure 1 .
2O [OOO3] Atypical standardized answer sheet for a multiple choice exam is shown in Figure 2. The examinee is required
to select one of the responses according to the directions provided with each item and fill in the appropriate circle on
the answer sheet. For instance, the correct answer to the question stated by stem 1 is choice B of the responses 3.
Thus, the circle designated 8 in Figure 2 corresponding to choice (b) is the correct answer to this item, i.e. question
1 3 should be filled in by the examinee as shown.
25 [OOO4] Generally, examinees register to take a particular test, by filling out a registration form and sending it to a test
processing center such as Educational Testing Service, Princeton, NJ by a specified registration date. A registration
form usually requires that an examinee provide information such as the examinee's name and address, test to be taken
and some related biographical information. After all of the registration forms have been received by the test adminis-
tration center, the examinee information such as name, address, some recipients background questions, etc., is proc-
3O essed. Each examinee is scheduled to take the test by assigning to that examinee a place and time at which the test
can be administered to that examinee. Typically, a number of examinees are scheduled to take the test at the same
time and same place to conserve on administrative costs. One or more test administrators will be present atthe locations
where the test is scheduled to be taken.
[OOO5] Test administrators are generally responsible for distributing the test material, providing instructions to the
35 examinees, monitoring any timing constraints required by the particular test and collecting the test material when the
testing time has ended or when the examinee has finished taking the test. After collecting the examinees' responses
and other test material, the administrator either directly or indirectly sends them back to the test processing facility, for
scoring and evaluation.
[OOO6] After all of the examinees' tests are graded, statistical and other processing may be provided for various
4O reasons. For instance, to assess one examinee's score, it is necessary to compare his or her score to those of other
examinees taking the same test. Another important reason to evaluate the test results for statistical purposes is to
create and update an informations bank containing the performance statistics of each item used or created for previous
tests. This information may then be used for the creation of future tests.
[OOO7] A goal of standardized testing is to construct a test efficiently for the purpose of measuring a skill, ability, etc.
45 Therefore, each test is constructed to conform to a test specification which defines the rules and/or constraints for
selecting the items. In constructing a test, test developers select items from pool of items so that the combination of
selected items satisfy the test specification.
[OOO8] A test is typically divided into sections of questions. The test specification generally defines the number of
items to be presented in the test, the number of test sections, the number of questions in each section, the time for
5O taking the test, and the allotted time for responding to all the items in each test section. The test specification also
specifies criteria for item selection. These are based on at least four item characteristics which include_. (1 ) item content,
e.g., mathematical questions relating to arithmetic, algebra, or geometry; (2) cross-information among items, e.g., more
than one item testing the same point; (3) number of items/set, i.e. a identification of a subset of items of a larger set;
and (4) statistical properties of items derived from pretesting, e.g. difficulty of the selected items.
55 [OOO9] In recent years, these methods for creating, delivering, administering, and scoring tests have been determined
to be inadequate. Due to the number of examinees taking standardized tests, the demand for developing new and
more diverse tests and a need to provide more flexibility in scheduling tests without sacrificing administration costs
and security have increased. 2
(PICTURE)
EP O 664 O41 B1
[OO1 O] One solution to these demands would be to automate the entire testing process. However, up until now only
a few attempts have been made to automate only portions of the testing process. Furthermore, these attempts are
limited in their ability to generate a variety of item types. They are not modular in their design to allow independent
replacement of software or hardware, nor do they provide security and integrity features required for a standardized
5 testing environment.
[OO1 1] There have been attempts to develop computerized tools for instructional purposes. These products, although
primarily geared to delivering instructional systems, often contain testing components as well. Some examples of in-
structional programs are available from Computer Curriculum Corp. , Computer Networking Specialists Inc. , Computer
Systems Research, DEGE M, Ideal Learning, Josten's Learning Corp. , New Century Education, Plato Educational Serv-
1O ices - TRO Inc. , Unisys - ICOPN System, Wasatch Education System, and Wicat Systems. Wasatch Courseware, for
instance, provides on-line tools, such as a notebook, a pop-up calculator, word processor, graphics tool, glossary, and
a database embedded into the lessons. Josten's Learning Corp. provides some flexibility in the hardware and software
available for executing lessons such as networked or non-networked systems, the use of third party software, and the
ability to operate its instructional system from a remote site. Ideal Learning has a management system which is also
15 capable of accommodating third party software, and its test scoring system can score tests which are generated by a
number of test developers including standardized tests. The DEGEM System is a networked system which is capable
of providing statistical data on student or class progress. Therefore, although some of these instructional programs
incorporate some features which could be utilized in an automated standardized computer-based testing system, none
of them provides a flexible and integrated system for developing, generating, delivering, administering and processing
2O computerized standardized tests.
[OO1 2] There are also a number of systems for computerizing parts of the test construction process. (See e.g. , a
review by Hsu and Sadock (1 985)). Perhaps the most comprehensive of these testing programs is the MicroCAT
System developed by Assessment Systems Corporation. The MicroCAT System comprises four primary subsystems,
one for each of development, examination, assessment, and conventional testing.
25 [OO1 3] Although MicroCAT has been noted for its comprehensiveness, it has been criticized for a number of limita-
tions. For instance, development of a test having a specification which does not match one of its predefined templates
requires a detailed understanding of MicroCAT's programming language. Its graphics tools are vey limited, and other
commercial drawing packages such as PC Paint cannot be substituted for MicroCat's graphics. Furthermore, there is
no on-line help available from either the test development system or from the examination system. Without an on-line
3O help facility, a system such as MicroCAT could not practically be used to deliver and administer standardized tests to
thousands of examinees each year. To use the MicroCAT assessment System, the test data must have been based
on tests which were generated only by MicroCAT's specifications. Furthermore, MicroCAT does not provide security
for examinee performance files nor does it provide integrity features to guard against power interruptions and the like.
[OO1 4] To accommodate standardized tests in computer based testing, there is a need for a comprehensive computer
35 based testing system which provides flexible test development and production, test administration and test delivery,
as well as preprocessing and postprocessing of item statistics and examinee performance. Such a system should
incorporate data integrity features, including system failure recovery and data security features. The design should be
modular and extensible so that substantially every hardware and software component can be modified or replaced
without affecting the functioning of the remainder of the system.
4O [OO1 5] US-A-5 OO2 491 discloses a computerized interactive instruction system in which a teacher is able to select
questions to be asked and monitor the students' responses in order to judge the effectiveness of the teaching.
[OO1 6] US-A-4 671 772 discloses a computerized management appraisal and feedback system in which users re-
sponses are used to compile a selection of video clips indicating a subject's strengths and weaknesses as a manager.
[OO1 7] US-A-5,O59, 1 27 discloses a computerized mastey testing system for the computerized implementation of
45 sequential testing in order to reduce test length without sacrificing mastery classification accuracy.
[OO1 8] Various aspects of the present invention are set out in the accompanying claims.
5O [OO1 9] The present invention will be better understood, and its numerous objects and advantages will become ap-
parent by reference to the following detailed description of the invention when taken in conjunction with the following
drawings.
[OO2O] Figure 1 is an example of written test questions and related directions.
[OO21] Figure 2 is a sample answer sheet used for paper and pencil tests.
55 [OO22] Figure 3 is a general overview of computer based testing facilities.
[OO23] Figure 4 is an interface diagram depicting the interfaces of each of the CBT (Computer Based Testing) systems
according to the present invention.
[OO24] Figure 5 is an interface diagram showing the subsystems of the test development system according to the
3
EP O 664 O41 B1
present invention.
[OO25] Figure 6 is an interface diagram showing the administrative system and test delivery system interfaces ac-
cording to the present invention.
[OO26] Figure 7 is an interface diagram showing the subsystems of the N DDS (Network Data Distribution System)
5 according to the present invention.
[OO27] Figure 8 is an information flow diagram for the ''TD/DC'' system.
[OO28] Figure 9 is a functional flow diagram of test production according to the present invention.
[OO29] Figure 1 O shows an example of a dialog box presented by the l PT (ltem Preparation Tool).
[OO3O] Figure 1 1 shows an example of some item components as viewed from the item preparation tool.
1O [OO31] Figure 1 2 shows the scrolling capability utilized by the item preparation tool.
[OO32] Figure 1 3 provides an example of selecting text in a stimulus by reverse video.
[OO33] Figure 1 4 shows dialog boxes presented by the l PT which prompt for response parameters.
[OO34] Figure 1 5 shows a dialog box presented by the l PT which prompts for key information.
[OO35] Figure 1 6 shows an example of an item having a stem referencing a demarcated portion of a stimulus.
15 [OO36] Figure 1 7 depicts an appropriate response to the item presented and shown in Figure 1 6.
[OO37] Figure 1 8 shows a second example of an item having a stem referencing the item's stimulus.
[OO38] Figure 1 9 depicts an appropriate response to the item presented and shown in Figure 1 8.
[OO39] Figures 2O and 21 show an example of a reference file replete with custom and interaction codes according
to the present invention.
2O [OO4O] Figure 22 shows the flow of items and keys from test development to test production services.
[OO41] Figure 23 shows a functional block diagram of the test preparation process according to the present invention.
[OO42] Figure 24 shows the relationship between the session script, test scripts, and units.
[OO43] Figure 25 provides a functional flow diagram of the test packaging process according to the present invention.
[OO44] Figure 26 is a block diagram showing the components of a test package.
25 [OO45] Figure 27 is a block diagram showing some item level components included in the presentation BLOB (Binay
Large Object).
[OO46] Figure 28 is a block diagram showing some test level components included in the presentation BLOB.
[OO47] Figure 29 is a block diagram showing some table components included in a SKM BLOB (Scoring Key Man-
agement).
3O [OO48] Figure 3O provides a functional block diagram of the test packaging process.
[OO49] Figure 31 shows a functional block diagram of the test delivey system according to present invention.
[OO5O] Figure 32 is a functional flow diagram of the test delivery process according to the present invention.
[OO51] Figure 33 shows some primary screen components of each screen presented during the test delivery.
[OO52] Figures 34(a)-(g) show examples of pane arrangements supported by the present invention.
35 [OO53] Figure 35 is an example of a directions screen.
[OO54] Figures 36(a)-(l) show examples of message screens supported by the present invention.
[OO55] Figures 37-39 provide some examples of Help Screens.
[OO56] Figure 4O provides an example of a review screen.
[OO57] Figure 41 provides a few examples of selectable testing controls supported by the present invention.
4O [OO58] Figures 42(a)-(b), 43, 44(a)-(c), 45-47 provide some examples of examinee interaction with items having
various response types.
[OO59] Figure 48 shows the data format of an examinee performance file according to the present invention.
[OO6O] Figure 49 shows possible data fields associated with the ''Start Session'' log event.
[OO61] Figure 5O shows possible data fields associated with the ''End Item'' log event.
45 [OO62] Figure 51 shows a functional flow diagram of the administrative process according to the present invention.
[OO63] Figure 52 shows some of the files on a workstation hard disk after it has been configured for computer based
testing.
[OO64] Figure 53 is a functional flow diagram describing the Close-of-Day Procedure.
[OO65] Figure 54 shows the data format of a log record according to the present invention.
5O [OO66] Figure 55 is a flowchart of the Main-Procedure of the Administrative Application.
[OO67] Figure 56A and 56B are flowcharts of the Start-System-Procedure of the Administrative Application.
[OO68] Figure 57 is an example of a system status screen.
[OO69] Figure 58 is a flowchart of the Logon-Procedure of the Administrative Application.
[OO7O] Figure 59 is an example of a sign-in screen provided by the Administrative Application.
55 [OO71] Figure 6O is a flowchart of the Process-State-Procedure of the Administrative Application.
[OO72] Figure 61 is a flowchart of the State-Null-Procedure of the Administrative Application.
[OO73] Figure 62 is a flowchart of the Change-Password-Procedure of the Administrative Application.
[OO74] Figure 63 is a flowchart of the State-Admin-Procedure of the Administrative Application.
4
EP O 664 O41 B1
[OO75] Figure 64 is an example of a screen showing the main menu of the Administrative System.
[OO76] Figures 65A and 65B are flowcharts of the State-Close-Procedure of the Administrative Application.
[OO77] Figure 66 is an example of a screen provided by the Administrative Application allowing an administrator to
enter test counts.
5 [OO78] Figure 67 is a flowchart of the State-Maint-Procedure of the Administrative Application.
[OO79] Figure 68 is a flowchart of the State-TDS-Procedure of the Administrative Application.
[OO8O] Figure 69 is an example of a screen provided by the Administrative Application when the testing session is
complete.
[OO81] Figure 7O is a flowchart of the State-Exit-Procedure of the Administrative Application.
1O [OO82] Figure 71 is a flowchart of the Menu-OpTest-Procedure of the Administrative Application.
[OO83] Figure 72 is a flowchart of the Menu-DemoTest-Procedure of the Administrative Application.
[OO84] Figures 73A and 73B are a flowchart of the Menu-TestCommon-Procedure of the Administrative Application.
[OO85] Figure 74 is an example of a screen provided by the Administrative application allowing an administrator to
enter examinee identification information.
15 [OO86] Figure 75 is an example of a screen provided by the Administrative Application allowing an administrator to
administer a test.
[OO87] Figure 76 is an example of an examinee confirmation screen.
[OO88] Figure 77 is an example of a screen provided by the Administrative Application allowing an administrator to
terminate the testing session or to edit the examinee information.
2O [OO89] Figure 78 is a flowchart of the Menu-RestartTest-Procedure of the Administrative Application.
[OO9O] Figure 79 is an example of a screen provided by the Administrative Application listing the available testing
programs.
[OO91] Figure 8O is an example of a screen provided by the Administrative Application listing the sessions that are
available for restart.
25 [OO92] Figure 81 is a flowchart of the Menu-CloseDay-Procedure of the Administrative Application.
[OO93] Figure 82 is a flowchart of the Menu-Exit-Procedure of the Administrative Application.
[OO94] Figures 83A and 83B are flowcharts of the Menu-Logon Maint-Procedure of the Administrative Application.
[OO95] Figure 84 is a flowchart of the Menu-CHG Password-Procedure of the Administrative Application.
[OO96] Figure 85 is a flowchart of the Stop-System-Procedure of the Administrative Application.
3O [OO97] Figure 86 is a flowchart of the Menu-About-Procedure of the Administrative Application.
[OO98] Figure 87 is a block diagram showing the inputs and outputs of the Network Data Distribution System (NDDS)
according to a preferred embodiment of the present invention. Figure 88 is an example of the Main Menu Screen for
the NDDS.
[OO99] Figure 89 is a preferred directoy structure of the NDDS.
35 [O1 OO] Figures 9OA through 9OC is a flowchart of a preferred procedure for processing CBT transmission files.
[O1 O1] Figure 91 is an example of a screen provided by the Administrative Application prompting the administrator
to enter the data disk.
[O1 O2] Figures 92A and 92B is a flowchart of a preferred procedure for processing the CBT Data Disks.
[O1 O3] Figure 93 is an example of a screen provided by the Administrative Application prompting the administrator
4O to insert the backup disks.
[O1 O4] Figures 94A and 94B is a flowchart of a preferred procedure for processing the CBT Backup Disks.
[O1 O5] Figure 95 is an example of the screen provided by the NDDS of an activity report selection menu.
[O1 O6] Figure 96 is a flowchart of a preferred procedure for implementing the activity reporting process according to
the present invention.
45 [O1 O7] Figure 97 is an example of a screen provided by the N DDS of an audit trail report selection menu.
[O1 O8] Figure 98 is a flowchart of a preferred procedure for implementing the audit trail reporting process.
[O1 O9] Figure 99 is an example of a screen provided by the NDDS of a Security/Event Log Report Selection Menu.
[O1 1 O] Figures 1 OOA through 1 OOF are examples of screens provided by the NDDS for carying out Computer Based
Testing Network (CBTN) processing according to a preferred embodiment of the present invention.
5O (PICTURE)
l. Computer Based Testing (CBT) System Overview
55 [O1 1 1] Referring to the drawings wherein like numerals represent like elements, there is illustrated in Figure 3 a
general overview of computer based testing. Computerized tests are developed at a central processing site 1 . Devel-
opment of a computerized test includes the creation of various data files by application specific software. The compu-
terized tests are packaged at the central processing site 1 and delivered to one or more test centers 2. Each test center
5
EP O 664 O41 B1
2 provides at least one workstation 3 on which a computerized test is administered to an examinee. In a preferred
embodiment, the workstation 3 is a personal computer equipped with a mouse.
[O112] A test center 2 may for example be located at a school or a dedicated test site. Generally, a test administrator
Iocated at the test center 2 loads the computerized test, data files and application software developed at the central
5 processing site 1 onto the hard disk of each workstation 3 at the test center 2. The administrator initiates the delivery
of a computerized test to an examinee who is scheduled to take the test. The examinee's responses to questions
presented by the test are preferably stored on the hard disk on each workstation 3 and are later preferably backed-up
by the administrator and transferred to the central processing site 1 for scoring and evaluation.
[O113] In Figure 3, one central processing site 1 , three test centers 2, and 9 workstations 3 apportioned among the
1O test centers 2 are shown. However, it should be understood, that any number test centers 2 and workstations 3 may
be used by the CBT system.
[O114] A block diagram of the CBT system is shown in Figure 4. The lines in the diagram demarcating each system
represent interfaces that pass information between the various systems which comprise the CBT system. The double
line separates those systems that reside at the test centers from those that reside at the central processing site. Those
15 systems shown within the double lines are the systems residing at the test centers, and those systems shown outside
the double lines are the systems residing at the central processing site.
[O115] Still referring to Figure 4, the CBT system comprises six separate systems. The Administrative System 1 4
provides substantially all administrative and control functions at a test center. The Test Delivery System 1 2 actually
presents questions and information tothe examinee at aworkstation. The Administrative System 1 4 initiates the delivery
2O of a test to an authorized examinee and secures the test by prohibiting access by any unauthorized person. Commu-
nication with the Test Delivery System 1 2 occurs through the Administrative System 1 4. The Network Data Distribution
System (NDDS) 1 8 receives data from the test centers 2 and distributes returned data to other systems at the central
processing site. The Test Development System lo provides substantially all functions necessaryto create test questions
and package computerized tests. The Preprocessing System 2O provides functions performed prior to the testing ses-
25 sion, such as registration. Such Preprocessing systems are typically custom designed for a particular test and are
provided commercially by various testing support companies such as Educational Testing Service, Psychological Cor-
poration, and American College Testing Service. Like the Preprocessing System 2O, Postprocessing systems are typ-
ically customized for each type of test and are provided commercially through Educational Testing Service, Psycho-
Iogical Corporation, and American College Testing Service. The Postprocessing System 1 6 provides functions per-
3O formed after the testing session, such as issuing official score reports or archiving examinee records.
[O116] A detailed block diagram of the Test Development System 1 O is shown in Figure 5. Five primary functions are
performed within the Test Development System 1 O, including test development/document creation (''TD/DC'') 62, an
item preparation system 6O, scoring and test key management (SKM) 66, computerized test preparation system 57
and test packaging 58. The Test Development System 1 O permits test developers to develop items and a Test Produc-
35 tion Staff (TPS) to computerize the items. It also supports the creation of test scripts. A test script is the electronic form
of a test. It provides option settings and configuration data necessary for delivering the test on a workstation. The Test
Development System 1 O also supports the creation of test keys, i.e., correct response to each item, and the packaging
of all components into one test.
[O117] Items are preferably written and created using the ''TD/DC'' (Test Development/Document Creation) system
4O 62. The ''TD/DC'' system 62 interfaces with the Item Preparation system 6O and with the scoring and key management
system 66 to computerizethe item content and key respectively. The Item Preparation System 6O is usedto computerize
the items for presentation by the Test Delivery System 1 2 and enter the computerized version of the item key which
differs depending on the item type. The Item Preparation System 6O prepares dataforthe scoring and key management
system 66 to communicate the computerized key information. The Item Preparation System 6O interfaces with the Test
45 Preparation System 57 to prepare the test scripts. The Test Packaging System 58 interfaces with the ltem Preparation
System 6O, the scoring and key management system 66, and the Test Preparation System 57 to obtain all of the item
and test components packaged into a computerized test. The NDDS Interface 56 transmits the computerized test from
the Test Packaging System 58 to the NDDS 1 8 as shown in Figure 4, for delivey to a test center 2 as shown in Figure
3. After an examinee has taken a computerized test, an postprocessing interface 64 with the Postprocessing System
5O 1 6 provides information about item keys to the ''TD/DC'' System 62 which uses this information to alter items or add
new items for use in subsequent tests.
[O118] Figure 6 shows a block diagram of the Administrative System 1 4 and Test Delivery System 1 2 and their
respective interfaces as shown in Figure 4. The Administrative System 1 4 permits test center administrators to control
delivey of tests, transmit results to a central processing site, and perform administrative functions such as backup of
55 data, item and software maintenance, and reporting. The Administrative System 1 4 fu_her prohibits access to the
computerized test by unauthorized persons.
[O119] An interface with the Network Data Distribution System (NDDS) 1 8 at the central processing site enables the
Administrative System 1 4 at a test center to send packaged examinee data and reports to a central processing site.
6
EP O 664 O41 B1
Data and software are sent from the central processing site to the test centers on diskettes.
[O12O] Still referring to Figure 6, the Test Delivey application Interface 26 is shown as having three specific interface
functions. First, the Administration system 1 4 can initiate a delivery of a computerized test and pass the necessary
information to the Test Delivery System 1 2. The Administration System 1 4 may also interact with the Test Delivey
5 System 1 2 for purposes such as terminating the testing session, processing examinee breaks, and timing essays, as
appropriate.
[O121] After termination of the test delivery, the Test Delivey System 1 2 transfers information such as examinee
performance data, return codes, and other processing data as appropriate to the Administration System 1 4 through
the Test Delivery Application Interface 26. The Administration System 1 4 then regains control of the workstation.
1O [O122] The Administrative Application 3O provides test center administrators with the ability to perform various func-
tions including_. controlling access to computerized tests and related data through levels of authorization and password
protection; entering and editing examinee identification information prior to the testing session; selecting the test to
administer; terminating tests; backing up examinee and administrative data; transmitting data to the central processing
site; changing passwords and adding or deleting administrator logon IDs; and reporting irregularity and activity data
15 to the central processing site.
[O123] A detailed block diagram of the Network Data Distribution System (NDDS) 1 8 with its interfaces is shown in
Figure 7. The Network Data Distribution System (NDDS) 1 8 provides substantially all necessary support functions for
the CBT system to control the network of test centers. The Test Center Administrative Application Interface 36 permits
the transfer of applications and computerized tests to the test centers and examinee records and reporting information
2O (data related to system errors and tesUworkstation security) from the test centers to the central processing site. The
Test Development Interface 42 provides the means by which new or revised computerized tests are sent from the Test
Development System 1 O to the NDDS 1 8. The NDDS 1 8 uses a Test Center Information Database 4O to determine
which test centers should be sent any new/revised tests, and to create reports from data received from the test centers.
The NDDS Processing component 44 receives data from test centers 2 via Administrative Application Interface 36,
25 checks it, sorts it, and processes it according to its type (program data, security data and reporting data). Reporting
data is used to create the necessary reports. Program data such as examinee records, are processed to consolidate
and reformat the information in a form suitable for postprocessing. The Distribution Interface 38 then distributes the
processed data to the Postprocessing System 1 6.
3O Il. Test Development System
A. Test DevelopmenUDo_ument Creation
[O124] In a preferred embodiment, test developers create tests at the central processing site. In computer based
35 testing, the tests are created by the test developers (TD) and are further processed and packaged by test production
staff (TPS). The ''TD/DC'' System which is developed by Educational Testing Service is preferably used to create the
test forms. It should be understood, however, that other test document creation systems could likewise be used to
create test forms for computer based testing. Therefore, although a detailed explanation of ''TD/DC'' is not required for
a description of the test development system, an understanding of its functional and procedural operation will aid in
4O understanding the test development system.
[O125] The ''TD/DC'' System 62 (Figure 5) is a fully automated system in and of itself. It consists of a central item
database and local personal computer based workstations. The central item database stores substantially all items
previously used on standardized exams as well as other items that have been created but not previously selected for
inclusion in an exam. Associated with each item stored in the central item database, is data stored in fields related to
45 the item's answer key, revisions it has undergone, a list of the test forms in which the item has previously appeared,
and statistical data indicating how the item performed at each previous administration. Other descriptive data fields
include information related to the item type (i.e., multiple choice or fill in the blank), the item's author, copyright infor-
mation, and both content and cognitive information specific to each testing program (e.g., SAT, GRE, etc.).
[O126] Every item is assigned a unique number called an accession number that identifies the item and all of its
5O associated data. The ''TD/DC'' system software allows items to be located by means of any of the datafields associated
with the item.
[O127] The central item database software allows test developers to access item information within the central item
database. This software supports the downloading of items and associated data to local workstations. Pools of items
may also be selecting and then downloaded. This software also enables test developers to bank, edit, and classify
55 features of the items stored in the central item database. Additionally, statistical information related to the use of an
item in an administration of a test is received by the central database software. Through a statistical feedback system,
this statistical data is added to the data stored in the central item database.
[O128] New items may be written by test developers at the local workstations. Software provided on the local work-
7
EP O 664 O41 B1
stations supports classifying, banking, and viewing these new items. With respect to the downloaded items, this soft-
ware also allows test developers to viewthose items and their associated data and permits the test developers to enter
and revise the statistical data.
[O129] Test developers also assemble draft tests on local workstations. The local workstations provide software
5 supporting item selection based on a number of criteria so that tests may be assembled to satisfy substantially any
test specification. When all of the items are selected to satisfy the test specification, these items and associated data
are assembled into what is referred to as a Worksheet.
[O13O] Test production for computer based testing, requires certain inputs from test developers and/or a test docu-
ment creation system. Figure 8 depicts the inputs provided by the test developer and the outputs generated by ''TD/
1O DC'' for use by TPS. For instance, ifthe ''TD/DC'' system is used, the information shown as offload files 74 and workfolder
76 is preferably provided to TPS. As described above, worksheets 72 are created by TD by downloading the selected
items and associated data. An offload program may then be invoked to copy the offload files, such as the item com-
ponent text (stem, stimulus, response, and directions), the response type, (e.g., multiple choice) response class (e.g.,
single response answer), the answer key and the item's accession number, onto a diskette. In preferred embodiments,
15 the TD also prepare a workfolder 76 containing information related to the computerized presentation of the items, and
graphics to be prepared as a computerized image and incorporated into the item presentation.
B. Test Produ_tion
2O 1 . Overview of Tegt Produ_tion
[O131] Test production comprises at least three primary functions, item preparation, test preparation and test pack-
aging. The Item Preparation System 6O as shown in Figure 5 is used by TPS to create an ''on screen'' version of the
items prepared. As described above, the test questions are prepared beforehand by test developers preferably using
25 the ''TD/DC'' System 62 or equivalent system. ltem text is edited until the content is satisfactoy to the test developers.
An offload program is used by the test developers to copy the selected items to a diskette which is sent with a work
folder of batch-related documents to the test production staff. The test production staff then makes a computer deliv-
erable image of the items in the form of files and prepares a test script for implementing the test. The test packaging
system 58 combines the item files with the test script to form the computerized test.
3O [O132] In a preferred embodiment, the Item Preparation system 6O comprises seven programs providing the functions
shown in the flow diagram of Figure 8. Table 1 below itemizes the programs used by the Item Preparation System 6O.
The table lists the purpose of each program, the program name, the operating environment in which the program
preferably is executed and the user during the test development process. ''WORD FOR Wl NDOWS'' and ''PAINT-
BRUSH'' are commercially available from Microsoft Corporation.
35 (PICTURE)
TABLE 1
4O
45 kTD - test developers
kkTPS - test production staff
5O kkkSYS - SysteMs
[O133] A functional flow diagram of the Test Production Process is shown in Figure 9. Assuming for purposes of
description that the system used to create the test document is the ''TD/DC'' system, items are first offloaded at 21 O
by test developers using the item offload function of the ''TD/DC'' system and are stored on a diskette. The offloaded
55 items and a corresponding workfolder are then transferred to the TPS. If graphics are to be shown with the item as
determined at 21 2, a computerized image of the graphics are prepared at 21 4 by TPS. TPS reads the items from the
diskette and separate them into component parts shown at 21 6 using the Item Element Generator (l EG). The text of
the item components can then be edited using a word processor at 21 8 such as ''WORD'' for example. Information
8
EP O 664 O41 B1
may be added to each component regarding, for instance, point size, font, leading, column arrangement, etc. during
word processing 21 8. Then, the items may be prepared at 22O using the Item Preparation Tool (IPT) to arrange the
components on the computer display and specify characteristics about each item (e.g., multiple choice, response type,
etc.). Finally, an electronic ''proofing'' copy of the items is returned to the test developers for review at 224. The test
5 developers maythen provide corrections orfinal approval. The proofing copy is preferably reviewable bytest developers
via a modified form of the Item Preparation Tool known as the Item View Tool (IVT).
[O134] TPS may iterate between word processing 21 8 and item preparation at 22O until satisfied with the results.
Likewise, the entire process from item offload 21 O through item review 224 may be repeated. Simple corrections can
be made by sending marked-up prints from the test developers to test production staff so that the most current version
1O of the test can be called to the screen again and edited according to the comments produced by the test developers.
[O135] Once final approval has been received from the test developers, a test is prepared as shown at 226 by a test
preparation tool based on information provided in the workfolder. Finally, TPAK packages all of the item files and test
scripts into a computerized test at 227.
15 2. ltem Preparatio_
a. Item Offload Program
[O136] The Item Offload function 21 O is executed by test developers on the ''TD/DC'' system using the item offload
2O program. This function e_racts data from the ''TD/DC'' system to be used as input for test production which ultimately
results in the creation of a computerized test. In the ''TD/DC'' system, a file which is known as the Worksheet and
described previously in Section Il (A), lists the accession numbers of items. Each Worksheet has a unique name which
is assigned by a test developer. Item offload 21 O creates a single file with all of the offloadable information for all items
in the Worksheet. The offload file's name is the same as the Worksheet name. Item offload 21 O opens the Worksheet,
25 reads each item pointed to by the accession number, and writes to an ASCll file the item's accession number, classi-
fication codes, rationale and item text.
[O137] Test production requires specific classification information. Thus, information contained in the Worksheet is
preferably categorized bythe test developers. One category may include item information. For instance, test developers
should provide codes that indicate how each part of the item's text is used for production purposes, e.g., the code
3O indicating that particular text is the stem, the response, or the directions associated with each item.
[O138] Another category of information which may be provided to test production is rationale information. Again, test
developers may insert codes and text related to the key description, number of responses required to properly answer
the question posed by the stem, a paraphrase summarizing the content of the item, and general remarks regarding
the appearance of each item.
35 [O139] A third category of information which could be provided to test production is classification profiles of the items
included in the Worksheet. The classification profiles may include codes identifying the item class, item type, item
structure, and information describing how the item should be presented to an examinee. The item class refers to its
response class and is indicative of whether a single response, multiple response, or free response is required by the
item. The item type code specifies its response type and is indicative of the type of response required to answer the
4O question posed by the item. For instance, the item type codes specifies selection of a value on a scale, selection of a
response with an ellipse or a box, insert text, or select a choice from a table. These and many other item types will be
described in detail below.
[O14O] The item presentation code specifies a predefined screen template. The screen templates indicate how many
panes the screen should be divided into for each item and the location of the panes. Additional codes are used to
45 specify which item information, i.e., stem, response, directions or stimulus, is to be placed in each pane and its position
within the pane. Similarly, the item presentation codes will be described in more detail below.
[O141] A fourth categoy of information which may be provided to test production is the stimulus formation for a set
of items. The stimulus formation code specifies the beginning of a stimulus to be shared by a set of items and the
presentation of the stimulus. When a stimulus is referenced by only a single item, the item is called a discrete item. In
5O the case of a discrete item, the stimulus information codes are preferably provided with the item information codes
rather than the stimulus information codes.
[O142] Prior to executing item offload 21 O, the test developer using the ''TD/DC'' system may insert tags in the item
and rationale text which are used by the Item Element Generator (IEG) program to separate the text into smaller
components. Tags in the item text are used to delineate the directions, stem and response components. Tags in the
55 rationale text are used to supply information about the key, number of required responses and a paraphrase summa-
rizing the item, as well as the rationale text. 9
EP O 664 O41 B1
b. Graphi_s
[O143] The work folder indicates whether graphics are required as shown at 21 2 in Figure 9. TPS may generate the
graphics using ''PAINTBRUSH'' as one example. The graphics files are named, for example, by adding an extension, .
5 Gnn, to the accession number of the item which contains the graphic to be presented. The ''nn'' is representative of a
number so that up to 99 graphics in this example may be associated with an item.
_. Item Element Generator (IEG)
1O [O144] The Item Element Generator (IEG) is a DOS program used by the TPS. The IEG is preferably written in
Microsoft C version 5.1 . The l EG separates the ASCll file created by the Item Offload program as shown at 21 6. The
individual items in the Offload file are separated from each other, then the items themselves are broken into components
and stored in files.
[O145] The component files are preferably named by tacking an extension onto an item accession number. The
15 extension specifies which piece of the item (i.e., stem, stimulus, response, directions, etc.) the file contains. Since the
accession number is used for the base name of a file, it is easy to locate all of an item components. Table 2 below lists
each component file with a predetermined extension. TABLE 2
2O (PICTURE)
25
3o [O146] A COntrOl file iS aISO generated by the IEG fOr eaCh item. A COntrOl file iS a maSter repOSitOry fOr an item'S
information. Specifically, the control file defines how these components are to appear as an item on a display during
test delivey. Minimally, a control file and a stem file are generated for each item. A directions file and a response file
may be generated if the item contains directions and response components. A reference file contains the stimulus
material and is generated only for items which belong to a set. Additionally, the control files of set members contain
35 the aCCeSSiOn nUmber Of the StimUIUS material.
[O147] The l EG also creates two log files during its execution and records errors in an error file. The error file and
Iog files are named by adding an extension to the base name of the input file, that is, the file generated during item
offload 21 O. Thus, for example, if the input file is named TEST, log or errorfiles are created by adding a unique extension
to TEST.
4o [O148] One Of the lOg fileS iS the Item ACCeSSiOn LiSt File, WhiCh COntainS the aCCeSSiOn nUmberS Of diSCrete itemS
and members of sets (excluding stimulus material) for which component files were produced. This file is named using
an input file name and .IAL extension such as inputfile.IAL. Accession numbers are preferably written to the file in
ASCll format, one per line, in the order in which they were processed.
[O149] The second log file created by l EG is the Batch Histoy File, which contains the accession numbers of all
45 itemS in the OfflOad file. ThiS file iS named USing an inpUt file name and .BHF eXtenSiOn SUCh aS inpUtfile.BHF. ACCeSSiOn
numbers are preferably written to the file in ASCll format, one per line, in the order in which they were processed.
Accession numbers of the set may be written to the file one after the other, starting with the stimulus material.
[O15O] The Error Log file is an ASCll file that logs errors encountered during l EG execution. This file is named inputfile.
ERR.
5o [O151] The .IAL and . BHF are preferably aIWayS generated. The . ERR file iS generated if errOrS are repOrted. Indi-
vidual component files are generated depending upon the item type and tags embedded in the item by test developers.
[O152] In a preferred embodiment, IEG uses a code conversion table to convert ''TD/DC'' classification codes to Item
Preparation codes. By using a separate file for this table, the l EG does not have to be recompiled and linked if the
codes are extended or altered in any way.
55 d. Item Preparation Tool
[O153] In a preferred embodiment, the Item Preparation Tool (l PT) is a Wl NDOWS-based application used by the
1 O
EP O 664 O41 B1
TPS. The IPT is also preferably written in Microsoft C version 5. 1 along with the Wl NDOWS Software Development
Kit. It is the electronic analog of tools with which to prepare the item image. Using the Item Preparation Tool, TPS can
process items received after the component parts are separated by the l EG.
[O154] After item preparation is started at 22O, test production staff can scan a list of items available for processing
5 from the item accession list file which is created by the l EG at the step 21 6. The list is generated by compiling all files
with the .STE extension in the item accession list file.
[O155] Each item having a .STE file extension is then edited and processed by test production staff to create a
computer deliverable form of the item which is developed by the test developers. In combination with the item prepa-
ration process 22O, test production staff can edit any of the item text components by word processing 21 8. After editing
1O the text of the desired component, test production staff may return to item preparation 22O. The revised text may then
be displayed by the l PT.
[O156] The IPT is preferably a menu-driven application. The lowest level menu options have dialog boxes. Dialog
boxes are typically used in a Wl NDOWS environment for prompting a user to input the data. Most dialog boxes typically
have two selectable buttons. One button labelled ''OK'' is selected by a user to exit the dialog box when he or she has
15 finished entering the data. The second button ''CANCEL'' allows the userto exit the dialog box without entering the data.
[O157] Table 3 below describes each of the menu options and the content of the dialog box presented to a user, i.e.
test production staff, after invoking the menu option in a preferred embodiment. It is well known in menu-driven appli-
cations to layer the menus. For instance, IPT provides eight main menu options in capital letters in Table 3 below, e.
g. Fl LE, Vl EW, PRESENTATlON, etc. One or more lower level menus may be invoked when one of these main menu
2O options is selected by the user. For instance, the menu selection listed in Table 3 such as ''PRESENTATlON/Compo-
nents/ Directions'' indicates that the user had selected the main menu option PRESENTATlON. Afirst lower level menu
was then provided, and the user selected ''Components.'' Subsequently, a second lower level menu was provided, and
the user selected the ''Directions'' option.
25 (PICTURE)
TABLE 3
3O
35
4O
45
5O
55
EP O 664 O41 B1
(PICTURE)
TABLE 3 (continued)
5
1O
15
2O
25
3O
35
4O
45
5O
55 1 2
EP O 664 O41 B1
(PICTURE)
TABLE 3 (continued)
5
1O
15 Detailed flowcharts and corresponding pseudo code of the l PT application is provided in Appendix A. However, the
following example will be provided for a more complete understanding of the use of the item preparation tool.
_o [O158] Figure 1 O shows a dialog box 23O generated by the Item Preparation Tool when a user selects by clicking the
menu option Fl LE/Open. The Fl LE menu option then lists several other menu options including an ''Open'' option. The
user then selects ''Open'' and the dialog box 23O as shown in Figure 9 is opened and prompts for the opening of a
particular item.
[O159] ''Open_.'' 228 shows which item is selected to be opened. ''Path_.'' 229 shows where on the hard drive these
_5 items are stored. ''Files'' 231 provide a selectable list of all the available items in that directory. ''Directories'' 233 provide
a selectable list of file directories on the hard drive. The ''Open'' 223 and ''Cancel'' 225 buttons may be executed after
the appropriate item is selected or to exit the ''Open'' option.
[O16O] Together Figures 1 1 and 1 2 present a screen-print of a simple reading passage. The top line 232 in Figure
1 1 reveals the name of the item identified by the item accession number. The second line 234 is a menu bar from which
3o item preparation functions can be initiated to construct the item identified in line 232. The left-hand box is a pane and
contains the reading passage 236. The reading passage 236 is the stimulus and is contained in the reference file for
the item, MHOOOOO1 . REF. The right-hand pane contains the directions 238 and the stem 24O which are stored in
separate files, but designed to be displayed in the same pane. The stem 24O in this instance indicates that the answer
should be entered by interacting with the reference file or reading passage 236. By inserting specific types of custom
35 codes referred to as interaction codes into the text of the reference file, an examinee can respond by selecting a
sentence in the reference file. The sentence will become highlighted on the screen as shown in Fig. 1 3 after it has
been selected. Line reference numbers 242 are found directly to the left of the text of the passage. They are preferably
included in graphic files and are not actual text.
[O161] A reading passage is often longer than the screen size allows. Figure 1 2 shows the scrolling feature of the
4o Item Preparation tool that allowsthe userto move through the passage by using the scroll mechanism 244 Iying between
the panes 246 and 248. The use and implementation of a scrolling feature are well known.
[O162] Figure 1 4 shows a dialog box 25O that prompts for response parameters after invoking the RESPONSE menu
option. These parameters set up the nature and functions of the response area. The ''Number of Req. Responses;''
field 247 indicates how many responses are necessary in order to answer the item. The ''Response Class'' box 249
45 indicates the general categoy of the response, i.e., single choice, multiple choice, or free response. The ''Response
Type'' box 251 indicates the specific form of the response. A description of the different response types is provided in
Section Ill below.
[O163] Further demarcation of the item is accomplished by entering information in a series of nested dialog boxes
as shown in Figure 1 5. For instance, if ''Multiple Choice'' box 239 is selected from the ''Response Type'' box 251 , then
5o a ''Multiple Choice'' box 252 is opened. The ''Number of Choices'' 241 may be computed by the Item Preparation Tool.
For example, in Figure 1 5, the ''Number of Choices'' 241 is set at eight. This number is based upon the amount of
specific interaction codes included in the response area. This number is important in error checking for codes because
it should reflect the exact number of available responses available, i.e., whether the number is less than the intended
number of responses, The Item Preparation Tool may indicate an error in coding. ''Block Set'' 243 specifies which set
55 of interaction codes are to be read by the software for a specific item. ''Indicator'' 245 prompts for different response
designs. Examples of indicators are also described in Section Ill below. ''Invert Choices'' 237 indicates whether the
response option should be highlighted by reverse video when selected. The component 253 permits test production
staff to enter which of the options is the correct choice. It should be understood that different response types require
13
EP O 664 O41 B1
different parameters than the example shown in Figures 1 2 and 1 3. The data entered by test production staff in the
dialog boxes shown in Figures 1 4 and 1 5 are stored in the item control file.
[O164] A different item is shown in Figure 1 6, although the same reading passage 236 in the reference file is used.
In this instance, part of the reading passage 236 is highlighted in reverse video 257 when the item is presented. Note
5 again the directions 254, stem 256, and response 258 are included in one pane. However, different response param-
eters have been set for this item, i.e., there are four options instead of eight. Interaction is set to the response file and
not to the reference file, ellipses are included rather than invert choices. Figure 1 7 shows the selection of a response
option 255 from the responses 258.
[O165] Figure 1 8 shows yet a third item which refers to the same reading passage 236. The stem 259 asks for a
1O word to be selected in the second paragraph. Although the item parameter dialogue box is not shown, item parameters
are set up in a third way so that there are 42 options (as many words as in the second paragraph). Interaction has
been switched back to the reading passage 236 or reference file, and the choices selected by an examinee will be
inverted. Figure 1 9 shows the correct answer at 261 after being selected.
15 e. Wo_d __o_eggi_g
[O166] In a preferred embodiment, the word processor is ''WORD FOR Wl NDOWS,'' which is available from Microsoft
Corporation. It is used by test production staff to edit component files produced by the l EG or to create completely new
items. Component files are edited for two purposes. The first purpose is to effect the appearance of the item text by
2O adding fonts, point size, bolding, etc. The second purpose is to insert ''Custom Codes'' before and after sections of text.
[O167] Component files are initially stored in ASCll by the l EG as described above, but they are converted to Micro-
soft's RTF format when they are saved in ''WORD FOR Wl NDOWS.'' Even though ''WORD FOR Wl NDOWS'' does the
conversions, the test production staff is responsible for selecting the correct format. After editing, text is written in RTF
format backtothe component file from which it came. Thus, a componentfile may contain either ASCll or RTF formatted
25 data, depending upon whether the file has been edited by ''WORD FOR Wl NDOWS.''
[O168] Now turning to the second reason for editing a component file, namely to add Custom Codes, literal strings
are inserted into the text and thus allow computerized features to be added to the test. In a preferred embodiment,
CUStom CodeS aIWayS Start With a '''i.''
[O169] Custom Codes belong to one of three classes_. 1 ) codes that stand alone, 2) codes followed by a parameter,
3O and 3) codes followed by additional data. ''Stand alone'' codes appear in the text by themselves. Their vey presence
is all the information conveyed by the code.
''Parameterized'' codes are distinguished from other classes of codes in that they are followed by a parameter enclosed
in square brackets (''['' and '']''). The parameter immediately follows the code without any intervening characters or
white space. ''Data'' codes are followed by other data. The data is arbitray text. To prevent conflicts with parameterized
35 codes, a single white space character is used to separate the code from the user-supplied data.
[O17O] Table 4 below summarizes some examples of Custom Codes and their use. Optional elements of parameters
are enclosed in curly braces (''('' and '')''). These codes are used to include a graphic in the component. The parameter
''nn'' is used to form part of the graphic file extension. Graphic files are named accession.Gnn, where accession is the
same as the base name of the file in which the graphic code appears. Thus, for example, if the graphic appears in a
4O stimulus component whose name is TD-OOO81 . REF, the graphic file name may be TD-OOO81 .Gnn.
(PICTURE)
TABLE 4
45
5O
55 14
EP O 664 O41 B1
(PICTURE)
TABLE 4 (continued)
5
1O
15
2O
25
3o [O171] Figure 2O is an example of the actual reference file in ''WORD FOR WINDOWS'' where it is manipulated and
replete with formatting and interaction codes. The control of the reference file shown in Figure 2O is based on the
sample items shown in Figures 1 O through 1 7. The name of the file is included in the first line of text. In this instance,
'''iACCESSlON[MHOOOOO1 ]'' iS the file name. The neXt CUStom Code on the firSt line iS an interaCtion Code, '''iPMC'' WhiCh
is a response code and indicates a ''Place Multiple Choice'' to be included in the passage. Since this code is not item
35 specific, it can be used once and be referenced by any number of items (in this instance, by items 1 and 3). The next
element, '''iGRL[O1 ],'' iS a Code that CalIS in a graphiCal image WhiCh eXiStS in a Separate file. ThiS Code indiCateS that
at this point in the file, a graphic should be placed to the left margin before any more text is included. The graphic in
this instance is the line reference numbers 5 through 2O. The graphical numbers 25 through 3O lay within a separate
graphical file, included after the word ''us'' in the third paragraph of the text of the passage.
4o [O172] One Can find the '''iGRL[O2]'' Code in FigUre 21 in the foUrth line of teXt. The Small arroW 26O after the graphiC
code is a ''WORD FOR WINDOWS'' formatting command. It is a tab marker which specifies the paragraph indent.
[O173] The neXt interaCtion Code, '''iBKS[1 ],'' iS a ''BloCk Start'' Code; it Will be folloWed by a '''iBKE[1 ],'' ''BloCk End''
code. These codes set the boundary around a specified option for a specific item. At this point, it is helpful to refer
back to Figure 1 5 and note that a ''Block Set'' number is identified for each item. Where there is more than one item
45 referring to the same area of text, separate Block codes may be included. In the passage there are also block codes
'''iBKS[2]'' and '''iBKE[2]'' indiCating option boUndarieS for SeleCting a reSponSe in the third eXample item aS ShoWn in
Figure 1 8 that uses the same area of text. Thus, only those portions of the reading passage 236 which are ''blocked''
by interaction codes which are related to a specified item will be active when that item is presented.
[O174] In the fourth and seventh lines of text in Figure 1 8 there is a iHCS[2]- _CE[2] code set, indicating a highlight
5o and center. This interaction code highlights the demarcated area of the reading passage 236 and centers it when the
item is presented by the item preparation tool, as shown in Figure 1 3.
[O175] Finally, in FigUre 21 , there iS a '''iHOR'' Code that prodUCeS a horiZontal line at thiS point in the teXt When read
by the item preparation tool and an '''i END'' Code, indiCating the end of the field of CUStom CodeS.
55 f. Translation
[O176] XLATE is one of the item preparation programs listed in Table 1 and compiles RTF format documents created
by ''WORD'' into a binay equivalent. Binary conversion speeds up the execution of the text display modules embedded
15
EP O 664 O41 B1
in the Item Preparation Tool and also results in a storage savings (mostly memoy savings).
[O177] XLATE is a system program which is generally not executed by test production staff from a command line,
menu pick, icon, etc. The Item Preparation Tool runs XLATE whenever it is determined that a component file has been
changed within ''WORD FOR WINDOWS.'' This is detected by comparing the date/time stamp of the .STE, . RSP, and .
5 DIR files of the item currently displayed by Item Preparation Tool with that of their binary equivalent the .STB, . RSB,
and . Dl B files. The binary file name is created from the component file name extension by changing the last letter of
the extension to ''B.'' For example, if the stem component's name is accession.STE, the binary file outputted by XLATE
will be accession.STB. If the binary file is older, its source equivalent must have been edited while the user was in the
''WORD FOR WINDOWS''. Thus, The Item Preparation Tool runs XLATE to update the binay file by translating the
1O source code.
[O178] During conversion, errors are preferably written to an error log file in ASCll format. The error file name is
created from the template XLATE???.ERR by substituting the component file extension for the question marks. Thus,
for example, if the component file is named accession.STE, the error file name will be XLATESTE.ERR.
15 g. Data _nterfa_eg and F_ow between Tegt Deve_op_ent and Tegt Produ_tion
[O179] Figure 22 presents a flow diagram of items and keys between TD 65O and TPS 66O. As previously stated in
Section Il. B., the test developers create and select items for a particular test preferably using the ''TD/DC'' system.
However, test items may be created and selected by any test document creation system or prepared by hand as long
2O as TPS 66O is provided with the information described in section ll.B.2.a. related to item offload. Three methods of
providing the information to TPS 66O are shown in Figure 22.
[O18O] The three methods are enumerated in Figure 22, by the numerals 1 , 2 and 3 located along the path lines.
The use of the ''TD/DC'' System 652 is enumerated as path 1 . If the key descriptions are prepared by a method other
than ''TD/DC'' 652, they may be written on paper and provided to TPS 66O via path 2. Additionally, the item text,
25 presentation information and classification information which are collectively referred to as the item description, may
also be determined by test developers who do not use the ''TD/DC'' system, but prepare this information on paper as
shown by path 3.
[O181] Test developers create and select the items to be included in a test as shown at 65O. If the test developers
use the ''TD/DC'' system, they execute item Offload to produce a diskette having the files containing the item description
3O and respective keys as shown at 652 and 658. The diskette is then sent to TPS as shown at 66O. If the files containing
the key information are not generated by the test developers using the ''TD/DC'' system, but rather by another method,
the test developers may provide a written key description to TPS 66O as shown at 654. Similarly, test developers may
prepare a written form of the item description shown at 656 and provide the written description to TPS at 66O. If graphics
are included in the written description at 656, a CBT artist will prepare computerized graphics files at 662 and 664.
35 These item graphics component files are also provided to TPS as shown at 66O.
[O182] If the item and key description are provided to TPS 66O via diskette created during item offload, TPS invokes
the item element generator at 666. The component files are separated as described above and filed in the an Item
Preparation (IP) database at 67O. If the item description or key description is presented to TPS 66O on paper, TPS
must manually enter the information using the IPT and word processing at 668. Then, the component files created by
4O TPS 66O are stored in the IP database 67O. Once all of the necessay component files are stored in the IP database
67O, TPS uses the IPT (ltem Preparation Tool) and word processing to prepare the computerized version of each item
at 668. Component files may be replaced in the l P database after being edited as shown at 67O and further processed
at 668 using IPT or word processing.
[O183] When TPS is satisfied with the computerized version of the item, the test developers may view the items as
45 they will be presented to an examinee as shown at 672 and 674 using the lVT. lf the test developers desire changes
to the items as presented, they can provide revision information to TPS Staff via paths 2 or 3 and the whole cycle may
be repeated until items are completed.
3. Test Preparation
5O a. Overview
[O184] Figure 23 shows the functions performed by TD and TPS and the software which is used to perform these
functions for preparing a computerized test. Test developers assemble the test as shown at 682. As shown at 686,
55 item selection is preferably automated (AIS) using the ''TD/DC'' system or an equivalent test document creation system.
Using ''TD/DC'', test developers enter the test specifications into the ''TD/DC'' system. Based on these specifications,
''TD/DC'' searches its central database for items which satisfy the test specification, e.g., 5O math questions, 25 of
which are algebra problems and 25 which are geomety problems. Then, the test developers review the items selected
16
EP O 664 O41 B1
by ''TD/DC'' for sensitivity and overlap constraints described in the background section. If the test developer decides
that the sensitivity or overlap constraints are not satisfied by the current selection of items, certain items may be des-
ignated to be replaced by another item from the database. In addition, test developers provide a test description spec-
ifying the directions, messages, timing of sections, number of sections of the test, etc. as shown at 692. If a computer
5 adaptive test (CAT) is to be run, test developers may run a computer adaptive test simulation at 684 which are known
to skilled test developers.
[O185] Using the Test Preparation Tool (TPT) and TOOLBOOK 696, TPS prepares the test level components as
shown at 7OO. TOOLBOOK is commercially available from Asymetrix Corporation. The test level components include
scripts 71 6, item table block sets 7O6, general information screens 7O8, direction screens 71 O, message screens 71 2,
1O and tutorial units 71 4. Each of the test components will be described in detail below.
[O186] As the components are prepared, the TPT stores them in a TPS networkdirectory 7O2. Then, the components
are entered into the TPS Production database 7O4. The components stored in the TPS Production database 7O4 will
be retrieved during test packaging which is described below.
[O187] A script consists of a series of files and further specifies the option settings and configuration data which the
15 Test Delivery Application (TDA) needs for operation. Option settings are specified for the Test Delivey System to
determine whether a certain feature is enabled for the test. Most option settings are simple yes/no declarations, but
some offer a limited set of choices (e.g., mouse speed_. slow, medium, fast). Configuration data is highly variable in-
formation such as the section name to be displayed in the Title Line during a test session or the list of items to be
displayed.
2O b. S_ripts
[O188] During test preparation, scripts 71 6 are prepared and combined with the items prepared during item prepa-
ration. Scripts control the sequence of events during a testing session. Two types of scripts 71 6 are preferably used_.
25 the session script 71 8 and one or more test scripts 72O. The session script 71 8 controls the order in which units within
the testing session are presented. Units provide specific services to the examinee, such as delivering a test or pre-
senting a score report. Just as the session script controls the session, the test script controls what is presented to the
examinee during the testing unit. Each testing unit may include one or more delivey units, which are separately timed
and scored subdivisions of a test. The system can dynamically select, or spiral, scripts and other test components so
3O that examinees are given what appear to be different tests. Figure 24 shows the relationship among session scripts
71 8, test scripts 72O, and units.
[O189] Some examples of units supported by the system are described in Table 5 below_.
TABLE 5
35 (PICTURE)
4O
45
5O
55 17
EP O 664 O41 B1
(PICTURE)
TABLE 5 (continued)
5
1O [O1 9O] Testing programs control the behavior and appearance of their tests through options that are enforced by the
15 Test Delivery Application. Table 6 below indicates the options that are available and the levels at which a testing program
can specify each option. The options, which are explained in the text following the table, are selectable at one or more
of the following levels_.
- test package (i.e., all instances of a particular test, such as GRE General),
_o - SeSSlOn,
- testing unit,
- delivery unit (section).
_5 (PICTURE)
TABLE 6
3O
35
4O
45 [O1 91] At the package level, options that are in effect for the entire package are defined in a Package Profile file.
Some examples include the program name, which may appear on the title line of the test. Another option may indicate
whether the program permits administrators to restart sessions after an examiner terminates the test. Additionally,
5o options may list the session scripts to deliver the test.
[O1 92] The session script is the second-level component of the testing package. It performs two primary functions_.
First, it specifies the Session Control Information, which defines the default options that are in effect for the entire
examinee testing session. Second, it controls the order in which units within the testing session are presented and the
options used to present them. The units that can be presented within a session script are_. General information screen
55 units, Tutorial units, Break units, Data collection units, Scoring and reporting units, and Testing units.
[O1 93] The session control information contains the default options in effect for the entire session. Control information
can be provided at multiple levels within the testing session. Thus, the control information provided at the session level
can be overridden by information that occurs later in the session. The information provided at the session level would
1 8
EP O 664 O41 B1
generally include the following_. Name - the session script name to be used by administrators in selecting a specific
session script from Administrative Application menus; Input device - the input device to be used during the session (e.
g. , mouse or keyboard); Color - the colors to be used during the session; Messages - program-specific messages to
override default messages during the session; Demo Script - indicates whether the script presents a demonstration or
5 operational test; Research Indicator -indicates whether the script presents a research pilot test; Special Timing - indi-
cates whether the script is standard or specially timed version.
[O1 94] The Gl S unit allows the incorporation of a single screen of information in the session. It may contain, for
example, the following information_. reference to the actual text and graphics that will be presented on the examinee's
screen; the type of dismissal_. automatic or manual; the time after which dismissal should occur.
1O [O1 95] The tutorial unit presents test familiarization materials to the examinee. It may contain, for example, the fol-
Iowing information_. reference to a tutorial, which is the familiarization information that will be presented on the exam-
inee's screen, e.g., ''Howto Use a Mouse'', ''Howto Scroll'', ''Howto Use the Testing Tools'', ''How to Answer''; information
that controls the content of the tutorial--the information varies depending on the tutorial selected. For example, for the
''How to Use the Testing Tools'' and ''How to Answer'' tutorials, the specific tools and item types to be covered must be
15 defined. The mouse and scrolling tutorials are generic--no content information is required when those tutorials are
selected; an indicator for first occurrence--this indicator applies only to the Testing Tools and How to Answer tutorials.
Introductory information is presented during the first occurrence of these tutorials and is omitted if they occur again
Iater in the session.
[O1 96] The break unit is used to implement a scheduled break within a session. It contains the following information_.
2O reference to a break procedure, which is the actual text and graphics that will be presented on the examinee's screen
during a break, along with the program that controls their presentation. It should be understood that multiple procedures
could be supported; the length of the break.
[O1 97] The data collection unit is used to obtain additional information from the examinee, such as demographic or
debriefing information. It is preferably implemented as a special instance of the testing unit, in which no scoring is done.
25 Like the testing unit, a data collection unit references a test script, which control as the sequence and options of the unit.
[O1 98] The scoring and reporting unit provides for scoring, and optionally reporting, the results of one or more testing
units delivered in a session. If the testing program selects a Display Scores option, it preferably displays all traditional
score types including raw, percent correct, converted and composite scores. If testing programs select the Cancel
Scores option, the examinee will be given the option of cancelling the scores.
3O [O1 99] The scoring and reporting unit preferably invokes the Educational Testing Services SKM (Scoring and Key
Management) routines to return the following information_. the score name for insertion into the score report, such as
''Reading'' or ''Antonyms''; the score type for insertion into the score report, such as ''number right,'' ''percentile, '' or
''converted score''; the score value, such as ''65O'' or ''passed''. It should be understood that any automated scoring
system which provides this information can be used or the information may be provided directly by a user.
35 [O2OO] The information needed to display a score report is preferably identical to that required for a message screen_.
reference to the actual text and graphics to be presented on the examinee's screen.
[O2O1] The testing unit presents a test, based on the contents of a test script that may have been selected at runtime.
The following units can be included within a testing unit_. general information screen unit; tutorial unit; break unit; delivey
unit, which delivers items to the examinee. This permits testing programs to interleave general information screens,
4O tutorials, and breaks with sections of a test. The testing unit contains the following information_. script selection mode
indicates whether dynamic runtime selection is to be used to select the test script; reference to a test script which
controls the sequence of events and options used during the testing unit. If dynamic runtime selection is to be used,
the reference is to a set of test scripts.
[O2O2] Like the session script, the test script performs two primary functions. First, it specifies the test and delivery
45 unit control information. Test control information defines the options that are in effect for the testing unit. Delivery unit
control information defines the options that are in effect for a particular delivery unit within a testing unit. It controls the
order in which units are presented within the testing unit and the options used to present them. The rules for presentation
of units are the same as those for the session script, except that an additional unit, the delivery unit, can be included
within a test script.
5O [O2O3] At least three delivey modes are preferably supported within one or more testing units_. linear, adaptive, and
essay. The test script references different components depending on the delivery mode of the test, but in all cases the
end result is a reference to a specific item or essay topic to present to the examinee. Multiple delivery units can be
used to organize the testing unit into sections, and each delivey unit can present a different mode of test.
[O2O4] The test control information includes the following information, which is in effect for the testing unit_. Logical
55 Name - the name used to associate the testing unit with a scoring specification provided by the SKM system or an
equivalent thereof; Directions - a reference to the text and/or graphics to be presented as general directions; Sections
- the number of sections within the test; create score data - whether scoring data for online scoring is to be created for
this testing unit; Message Overrides - any program-specific messages that are to replace the default messages within
1 9
EP O 664 O41 B1
the testing unit. The test control information can temporarily override the options specified in the session control infor-
mation.
[O2O5] In addition to the test control information, a test script can also contain delivey unit control information. The
delivey unit control information can be specified to change the options in effect for the duration of the section. The
5 following information is preferably included in the delivery unit control information_.
Type Indicator - whether this delivey unit delivers a test or a section; Directions - if the delivey unit is a section,
reference to the test and graphics of the section directions; Title Line Test - defines the name that will be used in
the title line_,
1O Mode - the delivery mode of the test or section, e.g., linear, adaptive, or essay;
Timing - timing options in effect for the test or section, including whether the test is untimed, timed in its entirety,
or timed by section, and whether timing begins when directions for the test or section are displayed or when the
first screen after the directions is displayed;
Scoring - whether scoring data for online scoring is to be created for the delivery unit;
15 Messages - program-specific messages to be used to replace the defaults during the test or section_,
Item Information - specifies how the times in the test are organized and where to find them. The contents of the
item information varies with the mode of the test being offered. For example, the item information for linear tests
is generally a reference to an item-by-item listing of the items to be presented, called an item table. Essay tests
may reference an essay procedure and a topic pool if more than one essay topic is provided for selection by the
2O examinee. Whether more than one essay is to be stored or only one is chosen for scoring is also defined. For
adaptive tests, reference to an adaptive algorithm and an item pool should be specified.
Testing Tools - the testing tools available during the test or section. Examples of testing tools are described in
detail in Section Ill(E);
Explicit Prompting - whether or not explicit prompting is to be used to make sure the examinee supplies exactly
25 the required number of responses_,
Must Answer - whether or not examinees should be allowed to move off an item without providing a response.
[O2O6] Detailed flowcharts and corresponding pseudo code for the TPT application are provided in Appendix B.
3O 4. Test Pa_kaging
[O2O7] After all of the items have been constructed for computer delivery by the test production staff and approved
by test developers and the scripts and tutorials have been created, the test production staff packages all of the relevant
files together using the Test Packaging Tool (TPAK) and the Score Key Management (SKM) system. In a preferred
35 embodiment, this process requires three steps. First, the test components are combined into a draft test package so
that the flow and presentation of the test may be reviewed. After the draft test package has been reviewed, the test
components are formed into a blue-line test package. After the blue-line test package undergoes successful quality
assurance tests, locks are applied so that data cannot be altered in the approved test packages. The final tests are
then distributed to test centers.
4O [O2O8] Aflowchart depicting the steps executed by TPAK and SKM to package a computerized test is shown in Figure
25. First, TPAK is used to create a delivery package at 6O1 . This step involves creating a presentation database which
incorporates the presentation information from the test scripts, creating a presentation parts lists which lists an ID for
each component used to create the presentation database and creating other files subsequently used by SKM. An
SKM database is created at 6O3 from the files generated by TPAK. These files preferably contain the item table de-
45 scribed above and item scoring information. The SKM database and the presentation database are then combined by
TPAK at 6O5 to produce installation files for distribution. After all quality assurance procedures have been performed
and satisfied, TPAK preferably applies a lock on all of the installation files as depicted at 6O7. After this lock is applied
a new test version should be created if changes to the test package are required.
[O2O9] Figure 26 shows the components of a final test package. Four primay groups of files are packaged together
5O to form the final test package. These four groups of files are the Profile and lndex files 8O1 , the Presentation Binary
Large Object (BLOB) 8O2, the SKM BLOB 8O3, and the Problem Item Notification (Pl N BLOB) 8O4.
[O21 O] Although the SKM BLOB 8O3 and Pl N BLOB 8O4 are shown in Figure 26, they are not necessary components
of the test package. However, they are preferably included in the test package. The SKM BLOB 8O3 should be included
if the examinee responses are to be scored at the testing center after the examinee has completed the test. If the SKM
55 BLOB 8O3 is not included in the test package, the examinee responses are scored at the central processing site by a
program specific (e.g., SAT, GRE) postprocessing system. The Pl N BLOB 8O4 is used to identify items which had been
included in the test but which are later determined either not to be scored or administered to the examinee. Thus,
although the Pl N designations provide a preferable feature, its inclusion in the final test package is not necessary. The
2O
EP O 664 O41 B1
details of the SKM and Pl N BLOBs will be described below.
[O211] The profile and index files 8O1 include the Package Parts List File (PPL) 8O5, the package profile file (PP)
8O6, and the BLOB Index Files 8O7. The PPL 8O5 contains a list of identification codes and version numbers, each l D
code and version number being associated with a component in the test package. The PP file 8O6 contains an identi-
5 fication of each test included in the test package. Although only one test is actually included in a test package, multiple
scripts may be provided for dynamic selection (spiralling) orforspecial conditions (e.g., untimed versions for examinees
with disabilities). Spiralling is atechnique in which test components (e.g., item substitution) are varied sothat examinees
taking the same test appear to be taking different tests. Spiralling inhibits cheating among examinees whose worksta-
tions are in close physical proximity because it increases the likelihood that each examinee will be interacting with a
1O visibly different test. The BLOB index files 8O7 provide an index for the Presentation BLOB 8O2, SKM BLOB 8O3, and
PIN BLOB 8O4 which are packaged into the computerized test. These indices function as a guide for locating specific
data within each of these BLOBs.
[O212] The presentation BLOB 8O2 includes item level components 8O8 and test level components 8O9. Examples
of the item level components 8O8 are shown in Figure 27 while examples of the test level components are shown in
15 Figure 27. Referring to Figure 26, the item level components 8O8 may include the item stimulus 82O, the item directions
821 , the item stem 822, the item response 823, and the item graphics 824. Each of these item level components has
been described in detail in conjunction with item preparation in Section Il.B.
[O213] Referring to Figure 27, examples of the test level components 8O9 are the test scripts 83O, general information
screens 831 , test level directions 832, message screens 833, and tutorial units 834. Each of the test level components
2O 8O9 has been described in detail in conjunction with test preparation or will be described in detail in conjunction with
test delivery in Section Ill.
[O214] Returning to Figure 27, The SKM BLOB 8O3 is preferably created by the scoring and key management ap-
plication and incorporated into the final test package. The SKM BLOB 8O3 preferably includes the Answer Keys 81 O,
scoring tables 81 1 , and scoring specifications 81 2. The answer keys 81 O provide the correct response or responses
25 to each item included in the test form. The scoring specification 81 2 provides information relating to how each item is
to be scored.
[O215] Figure 29 shows some examples of scoring tables which may be included in the SKM BLOB 8O3. The item
and custom item tables 84O, the item and custom item blocks 841 , conversion tables 842, Item Response Theory (IRT)
parameters 843, Theta estimation parameter tables 844, K-factortables 845, and Item weight tables 846 are preferably
3O included in the SKM BLOB 8O3. Since all of these tables are well known to those in the testing industry, a description
of these tables will not be provided here.
[O216] Returning again to Figure 26, the Pl N BLOB 8O4 includes pointers to pinned items 81 3 and the Do Not Score
(DNS) and Do Not Administer (DNA) flags 81 4. Some time after the computerized test has been prepared by TPS, it
may be determined that one or more items included in the test do not perform as intended, e.g., more than one response
35 would be correct although only one was intended. Thus, TD may designate these items and indicate whether the item
should not be scored or not be administered. SKM then creates the Pl N BLOB 8O4 with pointers and flags for each of
the items that has been found to misperform. When the test delivey system delivers the computerized test, it will not
present items flagged with a DNA and those items flagged with a DNS will not be scored.
[O217] Figure 3O shows the functions performed by TD, TPS, and SKM staff to implement the steps described above
4O with reference to Figure 25. Turning nowto Fig. 3O, as a part of test preparation, TD specifies the scoring specifications
and conversion tables at 776. In a preferred embodiment, SKM staff utilizes the automated SKM system developed
by Educational Testing Service as shown in Figure 3O. For instance, if the ETS SKM system or an equivalent automated
SKM system is utilized, the answer keys and tables may be retrieved from an SKM database at 77O and 756. The
scoring specifications and conversion tables are added to the data retrieved from the SKM database as shown at 758,
45 and an SKM BLOB 762 can then be created. TPS combines the test components, item components and BLOBs to
create a draft test package at 736 and 738 respectively. TD reviews the draft test package at 772 as it is delivered by
the Test Delivery Application at 778. If certain items do not perform as they should, TD identifies those items for the
SKM staff to create a Pl N BLOB 764 based on the information provided by TD. TPS adds the Pl N BLOB to the test
package and makes any other revisions the TD identified after reviewing the draft test package. When TD has author-
5O ized the test package at 78O, TPS prepares a blue line test package 74O and sets a level 2 lock on the test and item
Ievel components. A level 2 lock prevents modification to the locked components by unauthorized persons. After the
blue line test has been finally authorized by TD at 78O, TPS creates a set of data distribution disks at 742 and applies
a level 3 lockat 748. A level 3 lockvirtually eliminates the potential for any changes to be made to the computerized test.
[O218] Like the l PT and the TPT, TPAK is preferably a menu-driven application. Detailed flowcharts and correspond-
55 ing pseudo code of the TPAK application are provided in Appendix C.
[O219] In a preferred embodiment, a modified version of TPAK called ETPAK (Encypted TPAK) is executed. In
addition to verifying the presence of each of the required test files, ETPAK encypts at least the item files (e.g., .STE, .
REF, . RSP, and .CTL). After the required files have been packaged together, they may be transferred to a test center
21
EP O 664 O41 B1
at which the computerized test is administered and delivered to an examinee.
Ill. The Test Delivery System
5 A. Overview
[o22O] A block diagram of the Test Delivey System 1 2 is shown in Figure 31 . A test delivery application (TDA) 51 O
controls the test session, as directed by the test program 51 4, CBT files 51 6, and test delivery application data (TDA
data) 51 2. The test program 51 4 and CBT files 51 6 are administration system files and are preferably stored on the
1o work station or server (if workstations are networked via local area network) hard disk priorto delivery of a computerized
test to an examinee. Other files and applications such as the HELP facility 526 and the REVl EW facility 528 are also
preferably stored on the hard disk in advance. An examinee performance file 522 is created during each test session
to record an examinee's responses and other activity during the test session.
[O221] Although the Administration system will be described in detail below, a brief description is provided here as
15 it is relevant to the implementation of the Test Delivery system.
[o222] The center administrator uses a combination of manual and computer procedures to control operations and
deliver tests to examinees at the testing center. All of the files and applications shown in Figure 31 are sent to the
testing center in electronic form. All of these files can be loaded, on the workstations using standard set up procedureS.
The administrative application 51 1 of the Administrative system permits the administrator to initialize each workstation
2O at the start of the day, to sign an examinee onto a workstation, to start a testing session, and to close each station at
the end of the day. The center administrator is also responsible for performing backup procedures and transmitting the
examinee performance files 522 to the central processing site for scoring and evaluation.
[O223] Figure 32 provides a high level flow diagram of the test delivey procedure. In a preferred embodiment, when
the administrator completes the procedure to sign an examinee onto a workstation and selects a test at 5OO, the TeSt
25 Delivey Application is invoked. The Test Delivery Application reads the session script and executes the units it pre-
scribes. When the end of the session script is reached, the Test Delivery Application returns control tothe Administrative
Application.
B. Test Delivery Appli_ation Data Flow
3O [O224] Scripts define the sequence of tasks to be performed by the TDA 51 O as well as the information necessaY
to complete each task. The scripts define option settings, files containing program-specific text, and the items to be
displayed.
[O225] Although the following section provides a detailed description of the operation and use of the Test DeliveY
35 System and the presentation of various screens by the Test Delivery system, detailed flowcharts and correspondin9
pseudo code of the TDA are provided in Appendix D.
C. Title Line
4o [O226] The screen format during a delivery unit is preferably divided into three main sections; a title line 225O, an
item presentation area 2252, and a primary control area 2254 as illustrated in Figure 33. The title line 225O is preferably
presented as one solid gray barat the top of the screen. It should be understood that numerous other color combinationS
are possible. The title line 225O is capable of displaying various information relating to the test or taking the test. For
instance, the title line 225O may include the time remaining in the test or test section. Preferably, time is displayed
45 automatically although an examinee may optionally turn it off. In a preferred embodiment, remaining time is displayed
Ieft justified on the title line 225O in HH_.MM format until the last few minutes of the test. At that point, the display format
changes to MM_.SS and flashes for three seconds so that the examinee is alerted that the time remaining for taking
the test is nearly over.
[o227] Other information in the title line 225O may include the name of the computerized test and program-specified
5O text pertinent to what is being presented (e.g., section name). Information to help orient the examinee is also preferably
displayed in title line 225O. For instance, when an item or tutorial screen is displayed in the presentation area 2252,
the notation ''xx of yy'' or ''xx'' appears in title line 225O. The ''xx'' refers to the item number within the test or section,
or the screen within the tutorial. The ''yy'' indicates the total number of items in the test or section, or screens within
the tutorial. Additional orientation information to be provided to the examinee in the title line 225O may include the
55 descriptive word such as ''HELP'' ''REVl EW'' and ''Dl RECTlON'' to indicate a currently displayed screen. The ''HELP,''
''REVIEW'' and ''DIRECTlONS'' screens will be described below.
22
EP O 664 O41 B1
D. Presentation Area
[O228] The presentation area 2252 of the screen is used to display items screen such as the text and graphics and
non-item screens such as direction screens, message screens, HELP screens and REVl EW screens. Direction screens
5 are used to display directions for the test, section, and others. Message screens display information and examinee
options at transition points during the test session to control the flow of the test. Transition points indicate where a new
section or new item is to be displayed or when the test delivery application moves from an item screen to a non-item
screen. HELP screens and REVIEW screens respectively enable an examinee to interact with the HELP and REVIEW
facilities. These non-item screens will be described in more detail below.
1O 1 . S_reens
a. Item S_reens
15 [O229] Item screens are used to display items. Items can be mapped into the presentation area using one of a
predetermined number of standard templates. (See Table 3, Presentation/Template Menu in the description of the Item
Preparation System). In a preferred embodiment, the templates provide different combinations of one-, two-, or three-
pane arrangements resulting in seven possible templates such as those shown in Figures 34(a)-(g). The panes are
placed in the presentation area 252 like tiles which butt against each other but do not overlap. These arrangements
2O provide test developers with some flexibility in designing item layouts.
[O23O] In preferred embodiments, the text/graphics within a pane automatically becomes vertically scrollable if the
volume of information is larger than the pane size. Horizontal scrolling can also be supported although it is not required
for operation of the invention. A scrolling pane has a vertical (industry standard) scroll bar on the pane's right side. In
addition, a status bar may be placed at the top of the pane. The phrase ''Beginning,'' ''More Available,'' or ''End'' is
25 placed in the status bar and indicates the position of the currently displayed text within the pane as the examinee
interacts with the scroll bar. ''Beginning'' is displayed if the topmost information is visible in the pane. ''End'' is displayed
if the bottommost information is visible. ''More Available'' is displayed if neitherthe topmost nor bottommost information
is visible.
[O231] In preferred embodiments, the default size of a pane is 5OOXo of the presentation area. Thus, a two pane
3O arrangement cuts the presentation area into two equally sized panes -- either horizontally or vertically (see templates
shown in 34(b) and 34(e)). A three-pane arrangement is produced by simply cutting one of the panes that result from
a two-pane cut into two equally sized panes (see templates shown in Figures 34(c), (d), (f) and (g)). It should be
understood that the default size and ratios can be adjusted to create templates having panes of substantial size.
[O232] The components of an item can be mapped into any pane of a template, but preferably panes are not empty.
35 Additionally, components can share panes and can be placed in any order. Thus, for example, the stem and response
can be assigned to the same pane.
[O233] If a stimulus component exists, the test developer can elect to have a stimulus status bar placed at the top
of the pane containing the stimulus component. The status bar displays the status as the stimulus pane scrolls. The
stimulus status bar contains the phrase ''Questions xx to yy'' flushed left, where ''xx'' is the item number of the first item
4O to which the stimulus applies and ''yy'' is the last item to which it applies.
b. Dire_tion S_reens
[O234] Directions screens are typically used to display test, section, group and set directions. The directions may
45 contain text and/or graphics which are specified by the test script (i.e., declarations in test, section, group or set con-
figuration files). Scrolling is used to navigate through directions.
[O235] There are a number of ways to map the item directions component, i.e., accession. Dl R, into the item's pres-
entation area 252. For example, directions may be embedded above or below another component pane which may
contain stem, response, or stimulus. Alternatively, directions may be placed in a special directions pane that is inserted
5O below the title line 25O and above the template. Preferably, the directions pane fills the entire width of the presentation
area 252, and the vertical height of the directions pane is adjustable to fit in the presentation area along with any of
the template arrangements of Figure 34 (a)-(g). Vertical scrolling is preferably supported if there are more directions
than the pane can display. In a preferred embodiment, the size of the directions pane is specified and devoted to the
template. Template sizing rules are then applied as described above. Still further, item directions can be displayed
55 immediately before the item is displayed for the first time in a standard directions screen. _enerally, test and section
directions will only be displayed in a single directions screen.
[O236] Figure 35 is an example of a section directions screen. A single button, DISMISS Dl RECTlONS 265 dismisses
the directions screen when the examinee selects it. Once the examinee dismisses the directions, they are preferably
23
EP O 664 O41 B1
accessible through the HELP facility. If an examinee tries to dismiss the directions before scrolling to the end, a warning
message is preferably displayed. The message as shown in Figure 36(a) notifies the examinee that the directions
should be read completely and that they can later be retrieved through HELP.
[O237] In preferred embodiments, test directions are displayed as the first screen of the delivey unit. Test directions
5 notify the examinee, for example, the number of sections, misconduct notification, test administration instructions, and
break policy during the test session.
[O238] If the test contains sections, one set of section directions should be provided for each section. Section direc-
tions are preferably delivered at the start of each section. They include the number of items in the section, the time
allowed for answering the questions presented in each section, and the reference aids that will be required throughout
1O the section.
[O239] Group directions are used to introduce items of a like type; for example, analogies or antonyms. Group di-
rections are automatically displayed in a preferred embodiment upon displaying the first item of the group by the test
delivey application. Group directions are typically displayed once and eliminated thereafter. An optional paraphrase
may be associated with the group directions. The paraphrase is used to emphasize items on the REVl EW screen as
15 will be described below.
[O24O] Set directions are used to introduce a set of items that share the same stimulus material, e.g., an illustration
or reading passage. Set directions are also tied to specific items sharing the same stimulus material. Set directions
are preferably displayed when the first item of the set is to be displayed. Set directions are typically displayed once
and eliminated thereafter. An optional paraphrase may also be associated with the set directions. The paraphrase is
2O used to emphasize items on the REVl EW screen as described below.
_. Message S_reens
[O241] Message screens appear automatically in preferred embodiments at transition points and contain one or more
25 option boxes. The examinee should not continue interacting with the test until dismissing the message box by choosing
one of the options. When a message screen is displayed, clicking in any other location on the screen should be ignored.
[O242] Message screens consist of at least the message title line, text/graphics, and button icons. The title and icons
are fixed, but message files may specify the text and/or graphics that appear on message screens. Two types of
message screens are provided. One type pops up and overlays the center of the current screen. The second type
3O covers the entire display monitor. Examples of some possible message screens are shown in Figures 36 (a)-(l) and
described in Table 7 below.
(PICTURE)
TABLE 7
35
4O
45
5O
55 24
EP O 664 O41 B1
(PICTURE)
TABLE 7 (continued)
5
1O
15 d. HELP S_reens
2O [O243] The HELP screen is mapped intothe item presentation area wheneverthe examinee selects the HELP facility.
Figure 37 shows the format of a HELP Screen. Help buttons 28O are available from the HELP screen. The examinee
can display directions, scrolling instructions, etc. in the pane 281 by clicking on one of the buttons. The directions
include test, section, group, set, and item directions. Examinees use scrolling to navigate through HELP screens. For
Example, when the examinee selects the TEST Dl RECTlONS button 282, information will appear in pane 281 as
25 shown in Figure 37 and the selected TEST Dl RECTlONS button 282 will be grayed to indicate the current selection.
The selection of the grayed button will have no effect.
[O244] Referring to Figure 38, when the examinee selects the ''TESTING TOOLS'' button 283, a menu screen displays
all available testing tools for that particular section which are defined by the section configuration file of the test script.
When the examinee selects one of the tools, information about that tool is presented in pane 281 . For instance, Figure
3O 3g shows a sample screen which appears when an examinee requests help on the calculator testing tool.
[O245] In a preferred embodiment, the Help facility is context sensitive. If an examinee invokes HELP from a directions
screen, a message is displayed to further instruct the examinee as to how to proceed. If HELP is invoked from an item
screen and the following display items exist, the Help pane 281 will first display the group directions for that item, then
the set directions, and lastly the pop-up directions. If no directions exist for the item, a message informing the examinee
35 that the item screen includes all the necessay information will be displayed.
[O246] To exit the Help facility, an examinee selects the ''RETURN TO WHERE l WAS'' button 285 shown in Figure 39.
e. REVIEW S_reens
4O [O24_] The Review screen is mapped into the presentation area wheneverthe examinee selects the REVl EWfacility.
Figure 4O shows that the screen presentation REVl EW buttons 286 are available to the examinee from each Review
screen. The top pane 287 contains directions on how to conduct the review. The Review pane 288 is used to display
group, set, and item paraphrases (if provided), plus item status. Item status consists of a phrase and possibly a check
mark. In a preferred embodiment, the status phrase can specify ''Not Answered,'' ''Answered,'' or ''Not Seen.'' The check
45 mark indicates whether the item was marked for review by the examinee during the test session.
[O248] Scrolling is preferably supported in the Review pane 288 if the information cannot be presented on a single
screen. Upon invoking the Review facility, the item from which REVl EW was invoked is highlighted. The examinee can
highlight a different item and then select the ''GO TO QUESTlON'' button 289 to review that item. At any point, the
examinee can select the ''RETURN TO WHERE l WAS'' button 29O to return to the point from which the REVl EW
5O function was invoked.
[O249] In a further preferred embodiment, when the examinee clicks on one of the paraphrases and if the examinee
has not previously seen the group or set directions, they are displayed. Thereafter, or if the examinee has already seen
the group or set directions, clicking on the paraphrase brings up the first item of the group or set. If the REVl EWfunction
is invoked from a directions screen, the examinee is returned to the directions screen by selecting the ''RETURN TO
55 WHERE l WAS'' button 2gO. However, when the examinee moves to an item, the directions screen from which REVl EW
is invoked is considered to have been dismissed. Once directions are dismissed, they are still preferably accessible
through the HELP facility. 25
EP O 664 O41 B1
E. Control Area
[O25O] The primay control area 2254 preferably provides testing tools for giving the examinee a degree of control
over the testing session. In a preferred embodiment, there are ten testing tools (also referred to as ''primary controls'').
5 Each tool has its own icon. Icons are pictorial representation of a function available to a user which can be activated
by selecting that icon. The icons for each of these testing tools are shown in Figure 41 .
[O251] Referring to Figure 41 , the NEXT icon 2255, the PREV icon 2256, the REVIEW icon 226O and HELP icon
2257 can be used by the examinee to move from one screen to another screen. When the NEXT icon 2255 is selected,
the examinee can move on to the next screen. Selecting the PREV icon 2256 enables the examinee to move back to
1O the previous screen. The HELP icon 2257 can be selected by the examinee to invoke the HELP facility. When HELP
is invoked, the examinee moves to a HELP screen to retrieve previously presented direction and information about
topics covered in the tutorials. The examinee is returned to the screen from which HELP was invoked when the Help
screen is exited.
[O252] The MARK icon 2259 enables the examinee to mark an item for review. In a preferred embodiment, both
15 answered and unanswered items can be marked. A marked item is indicated on an item screen by displaying a check-
mark in the MARK icon 2259. The checkmark may also appear next to the marked item in the Review screen when
the REVl EW facility is invoked after an item has been marked. The examinee can unmark an item by clicking on the
MARK icon 2259 a second time. However, preferably the examinee need not unmark items in order to leave a section.
[O253] The REVl EW icon 226O, when selected, presents the review screen to the examinee listing the items in the
2O section in the order they were presented to the examinee, along with any group or set paraphrase associated with item
and an indication of whether the item has been marked by the examinee from the item screen. The examinee then has
the ability to go directly to any item in a test section by clicking, as described above.
[O254] Preferably, the examinee can invoke the REVl EW facility from any item screen, and the REVl EW screen will
display the status of all the items in the section regardless of whether all of the items had been presented. In a further
25 preferred embodiment, the examinee may skip some items by advancing to a subsequent item.
[O255] The ERASE icon 2258 enables the examinee to reset all selected choices for the current item to their original
state. The Tl ME icon 2261 allows the examinee to turn on and off the remaining time display in the title line 225O. The
EXIT icon 2262 allows the examinee to leave the current section of the test. The QUIT icon 2263 allows the examinee
to quit the test. The CALC icon 2264 allows the examinee to use on-screen calculator.
3O [O256] A testing tool is said to exist if it appears on every screen. The existence of each of the depicted icons; PREV,
CALC, QUIT, EXIT, Tl ME, REVIEW, MARK and ERASE is specified by each test script or the section configuration file.
The NEXT and HELP tools preferably exist in all tests. Testing tools that do not exist should not appear on the screen,
and in a preferred embodiment, the location of the remaining tools is adjusted to close any gaps left by non-existent
tools.
35 [O257] Test scripts can define the existence of tools to limit the ways in which examinees can navigate through the
test. For example, a program can define a forward-progression-only test by eliminating the PREV and REVl EW tools.
[O258] A testing tool is said to be available if it exists and can be used. Preferably, a testing tool is displayed in black
if available and in gray when it is not. For instance, in preferred embodiments, the NEXT icon 2255, the PREV icon
2256, ERASE icon 2258, MARK icon 2259, and the CALC icon 2264 are available only from item screens. However,
4O the HELP icon 2257 is available from all screens. The REVl EW icon 226O is available from item screens and group or
set directions screens while the Tl ME icon 2261 may or may not be available from directions screens depending upon
whether the section configuration file indicates timing is to start before or after the presentation of directions.
F. E_aminee Intera_tion
45 [O259] An item response is defined by its class and type as provided by the test developers and implemented by the
test production staff as described above with reference to Figures 1 3 and 1 5. The following will describe how an ex-
aminee can interact with the workstation to respond to each item according to its response class and type.
[O26O] As described previously, there are three response classes; single selection, multiple selection and free re-
5O sponse. Single selection requires that the examinee select only one answer. Multiple selection requires that the exam-
inee select more than one answer, e.g., ''select all that apply'' or ''select the two best.'' Free response requires the
examinee to enter a numeric expression or value, or text via the keyboard coupled to the workstation.
[O261] Depending upon the selected response class, implicit or explicit prompting will be given by the test delivery
application. Implicit prompting refers to a feature that will automatically de-select a first selected answer if an examinee
55 subsequently selects another answer for the same item. For instance, if the examinee selects answer (a) and then
changes his or her mind and selects answer (b), answer (a) will automatically be deselected by the implicit prompting
feature. In preferred embodiments, implicit prompting will be used for those items having a single selection response
class. However, other implicit prompting rules may be enforced for items having a free response class designation.
26
EP O 664 O41 B1
For instance, if the response requires the examinee to enter a numeric value or a fraction, implicit prompting will cause
the test delivery application to either ignore all alphabetic characters entered by the examinee.
[O262] Explicit prompting may be specified by the section configuration file. One purpose of explicit prompting is to
ensure the examinee supplies exactly the required number of responses. Explicit prompting, if elected, only applies to
5 items having a control file specifying the number of required responses, e.g., Multiple Selection. If explicit prompting
is specified, preferably examinees can complete an item only if they have supplied all required responses. In such a
preferred embodiment, the explicit prompting option will be enforced by the NEXT, PREV, REVl EW, and EXIT testing
tools.
[O263] As described previously, test developers select a response type for each item, and test production staff im-
1O plement this response type. The following will provide a few examples describing how an a examinee would interact
with the workstation to respond to items having various response type.
[O264] One of the most common response types is multiple choice. According to the present invention, multiple
choice may be implemented by requesting that the examinee select one or more choices from a list of choices presented
to the examinee or request that the examinee highlight one or more of the choices. If an indicator is presented along
15 with each listed choice, the examinee is requested to select the indicator associated with the choice which the examinee
desires to select. It should be understood that numerous variations of indicators are possible, e.g. ovals, ellipses, semi-
circles, etc. To select the indicator using the workstation described above, the examinee moves the mouse until the
cursor is positioned on the indicator that the examinee wishes to select. Then, the examinee depresses a button on
the mouse causing the oval to be darkened. It also should be understood that the indicator could be marked by dis-
2O playing an ''X,'' check mark or any other appropriate designation. Figures 42(a) is an example of the use of oval indi-
cators in which ''choice two'' was selected by the examinee. Figure 42(b) shows the choices that are selected by the
examinee. The selected choices have an ''x'' placed in the indicator box which is located next to each corresponding
response.
[O265] Alternatively, if the examinee is requested to select one or more choices by highlighting, the examinee places
25 the cursor on each desired choice by moving the mouse accordingly, and then depressing a button on the mouse. The
selected choices are then highlighted by reverse video as shown in Figure 43.
[O266] Another response type supported by the CBT system is the selection of a choice or choices on a scale pre-
sented tothe examinee with an item. Examples of possible scales include horizontal, vertical, semi-circular and circular
number lines. The number lines typically are accompanied by markings with a numeric value displayed adjacent to
3O some of the markings. The examinee can respond to an item that references such a scale by placing an arrow at a
Iocation adjacent to the scale or by filling in a desired portion for a circular scale. For example, refer to Figures 44(a),
(b), and (c) illustrating the placement of an arrow. To place the arrow at the desired location, the examinee moves the
mouse until the cursor is on the desired location and then depresses a button on the mouse. If a circular scale such
as the one shown in Fig. 45 is used, the examinee can move the cursor via mouse to the portion desired to be filled
35 in and depresses a button on the mouse. The portion is then filled in by reverse video as shown in Fig. 45.
[O267] Another response type supported by the CBT system is a bar graph or histogram such as the one shown in
Fig. 46. It should be understood that a bar graph is a grid formed by a horizontal axis and vertical axis with tic marks
or reference lines demarcating cells. In preferred embodiments, examinees are asked to extend one or more of the
bars on the bar graph in response to a question presented. To extend the bars, the examinee clicks on the mouse
4O button after positioning the cursor at the desired location. For instance, assume the examinee is presented the bar
graph shown in Fig. 46 without the bar shown for year '85. If the examinee is requested to fill in a bar showing the
correct number of units produced in '85, assuming that 8,OOO units was the correct answer, when the examinee clicks
the mouse at the location indicated at 295, a bar is displayed from the base of the graph to the line corresponding to
8,OOO units. Alternatively, one or more movable bars may be displayed. In such a case, the examinee may select one
45 of the bars by clicking the mouse when the cursor is positioned on the bar, moving the cursor to the location where the
examinee intends to move the selected bar and then releasing the button.
[O268] As described previously, the test developers determine what information should be provided with the bar
graph and specify all related parameters. For instance, test developers mayspecifyatitle and its placement; the number
of bars to be presented or added by the examinee; the size, shape and orientation of the bars; whether the bar is to
5O be movable_, the size and orientation of tic marks and reference lines defining the grid_, and any grid labels. The test
production staff implement these parameters by creating the required graphics and inserting the appropriate custom
codes as described in Section Il herein. Additionally, test production staff inserts interaction codes in each cell of the
grid which specifies whether or not the cell will be selectable by the examinee.
[O269] Another response type requires the examinee to order a list of choices into response fields or to match two
55 or more choices to possible response fields. This response type requires a two-step process to respond. First, the
examinee must select a choice for placement by pointing and clicking on a choice; the area containing the choices is
called the source. Next, the examinee must point and click on the response field in which it should be placed; the area
for placement of the selected choice is called the target. Placement will be indicated by copying the selected choice's
27
EP O 664 O41 B1
text/graphics into the target area. Clicking on a target area that has already been used erases that selection.
[O27O] The test developers specify whether a choice can be used more than once in the item. If the option to prevent
multiple use is elected and the test script also elects the explicit prompting option, the examinee will be informed that
he or she used a choice more than once and will not be permitted to move off the item until he or she responds as
5 directed.
[O271] An insert text response type may also be supported by the CBT system. This response type applies to items
in which the examinee is required to insert a block of text (word, sentence, etc.) into a reading passage. In a preferred
embodiment, the possible placement positions in the reading passage are indicated with black boxes. Examinees click
on the box where they wish to insert the text. When this occurs, the text is duplicated at the selected position in the
1O reading passage on a black background. The test developers specify the possible insertion point, and the test produc-
tion staff implements such specifications by inserting custom codes into the reference file.
[O272] A zone response type may also be supported by the CBT system. In zone response type items, the examinee
is required to select choices that are placed at various locations (referred to as zones) on the screen. This response
type is most effective when the test developer wants an examinee to select choices such as cities on a map or objects
15 of an illustration. An example is shown in Figure 4_. A map of the United States is displayed with all possible choices.
Possible choices may be identifiable by being presented with a rectangular boxcentered around each selectable choice.
A selected choice may be displayed by darkening the entire box by reverse video.
[O273] In a numeric entry item, examinees must enter a number to answer the question. Preferably, a box will be
provided for the examinee to enter a response. Examinees may enter their answer via the keyboard or by transferring
2O a result from the calculator display if the CALC testing tool is available for the item.
[O274] A fraction response type item is one in which the examinee must enter one or two numbers numerator/de-
nominator). In a preferred embodiment, boxes are provided for the response(s). Examinees preferably first click in one
of the two boxes to select it and then enter their answer via the keyboard. Their answer appears in the selected box.
[O275] Essay items are also supported by the CBT system. Examinees enter free-form text as if they were using a
25 word processor. Thus, the examinee enters the text via the keyboard.
G. The E_aminee Performan_e File
[O276] During the testing session, the test delivey application generates log records which are recorded in an ex-
3O aminee performance file. The examinee performance file is the outside world link to what happened during the exam-
inee's testing session. One performance file is generated for each test session. The performance file is created when
the administrative application initiates the examinee sign-on procedure. Each system event thereafter causes a log
record to be created and written to the performance file.
[O277] A preferred structure ofthe log record is shown in Fig. 48. In this preferred structure, each log record comprises
35 a standard header 4OO and a data area 4O2. The DATA LENGTH field 4O6 provides the number of bytes contained
within the log record. The SEQ# field 4O8 preferably begins at ''1 '' and increments each time a log record is written to
the performance file. The CHECKSUM field 41 O provides a checksum value resulting from executing a checksum
routine on all of the data fields contained in the log file. The Tl MESTAMP field 41 2 records the time when the log record
is written to the performance file.
4O [O278] Examples of events recorded in the examinee performance file are listed in Table 8 below.
(PICTURE)
TABLE 8
45
5O
55 28
EP O 664 O41 B1
(PICTURE)
TABLE 8 (continued)
5
1O
15
2O
25
3O [O279] Each event is preferably assigned an event code which is written in the EVENT CODE field 4O4 of the header
4OO. The data fields written in the data area 4O2 vary depending on the event being recorded. Thus, the event code
is used to identify the data fields written in the data area 4O2 of the log record. Other events that can be recorded in
35 ihe exam__nee pe_ormance f___e __nc_ude Siari/End _enera_ _nformai__on Screen, SiaruEnd Rev__ew, Siari/End D__reci__ons,
or Start/End Scoring and Reporting Unit.
[O28O] Center code is a code which uniquely identifies the test center where the examinee is taking the test.
[O281] The data fields of the Start session event data area 4O2 are shown in Fig. 49. In a preferred embodiment, the
data fields comprise two header field 41 4 and a number of data fields 41 6, 41 8, 42O, 422, 424, 426 and 428. The
4O Cenier Code f__e_d 41 6 __deni__f__es ihe iesi cenier. The worksiai__on _ f__e_d 41 8 un__que_y __deni__f__es a worksiai__on ai ihe iesi
center on which the test is delivered to the examinee. The Administrator Name field 42O specifies the administrator
who performs the sign-on procedure and initiates the test delivery application. The package control id 422 identifies
the current version of the testing program's package, and the software versions 424 identify the versions of Score Key
Management, Administrative, and Test Delivery software used for the testing session This information is used in the
45 eveni an exam._nee sess._on musi be dup_._caied, for examp_e, io reproduce a sysiem ërror The exam._nee ._nformai._on
426 may vary for different testing programs. It typically includes a registration id, the exam' inee's name, date of birth,
social security number, and indicators if the examinee walked into the center without scheduling an appointment or
required special testing conditions, for example due to a physical disability. The session information 428 identifies the
session script used for the test This script may have been selected from a set of session scripts forthe test via dynamic
5O runi._me se_eci._on _i a_so ._nc_udes a sess._on number ._nd._cai._ve of ihe number of i._mes a iesi._ng sess._on ._s ._n._i._aied for
an examinee. It t'he testing session must be restarted, e.g., due to a loss of power, a session number field will be
incremented.
[O282] The End Session event has one data field having information indicating how the session was ended. For
instance, the session could be ended via QUIT icon selection, by test center administrator interaction, or by a system
55 fa.__ure or resiari
[O283] Severa'l events do not require any datafields. These events include the Restart Session event, the end tutorial
event, the calculator events, the start and end HELP events, and the start break event. The timestamps recorded in
29
EP O 664 O41 B1
the header 4OO of a log record created upon one of these events is sufficient to convey all of the required information
needed by the postprocessing system.
[O284] The item number of each item as it is presented to the examinee is recorded in a log record upon each start
item event. When the examinee responds to an item and moves to the next screen, data is written to an end item event
5 log record. The data fields provided by a log record created upon the End Item event are shown in Fig. 5O.
[O285] The termination type field 43O identifies how the end item event occurred. Some possible mechanisms for
terminating delivery or presentation of an item include, moving to another screen via NEXT or PREV icons, moving to
a different item via the REVl EW facility, or by any of the means used to terminate a testing session (e.g., EXIT, QUIT,
etc.). The ''Marked'' field 432 provides information indicative of whether or not the examinee marked the item before
1O moving to a different item or screen. The ''item processing information'' field 434 provides the number of times an
examinee has clicked a mouse button during the item visit, the computer working time elapsed while the item was
being displayed, and the seconds remaining in the test at the point of the End Item event. The ''response type'' field
436 provides a value indicative of the response type associated with the item. The ''score type'' field 438 provides a
value indicating the scoring rule associated with the item.
15 [O286] The next three fields of the data area 4O2 of the end item log file relate to the examinees selected responses.
In a preferred embodiment, two types of response data can be provided. The first type refers to response types for
which the examinee is instructed to select at least one response or move or alter features presented to the examinee.
The second type of response data relates to items having a free response type, such as a numeric response. The
''response data format'' field 44O indicates which response data type the response data is stored in the log record. This
2O field may also indicate that no response data was provided. The ''response count'' field 442 provides a value indicating
the number of times the response data is repeated. The ''response data'' field 444 provides the examinee's response
or responses. Table 9 below explains a preferred format of the ''response data'' field 444 when the item has the first
response data type. If an item has the second response data type then the examinee's entered response is stored in
the ''response data'' field 444.
25 (PICTURE)
TABLE 9
3O
35
4O
45
5O
55 3O
EP O 664 O41 B1
(PICTURE)
TABLE 9 (continued)
5
1O IV. The Test Administration System
A. Fun_tional Overview
15 [O287] An administrative application and various administrative files manipulated by the administrative application
make up the administrative system.
[O288] A functional flow diagram of the functions implemented by the administrative system is shown in Figure 51 .
System installation 3OO is provided at each workstation on which a test is to be taken. In a preferred embodiment, the
administrative application can be run in environments with local area networks of workstations and a server or stan-
_o dalone workstations with hard disk storage devices. In local area network environments, the workstations may have
hard disk storage devices or be diskless workstations that store all information on the server. One station in each local
area network center is designated the ''master'' station, from which all the stations in the network are started, closed,
and maintained. The computerized tests are thus stored on the hard disk or the server in advance of a scheduled test
by the system installation 3OO.
_5 [O289] System installation 3OO may be implemented in a number of ways. For instance, the computer test may be
transported from the central processing site on floppy disks and loaded by the test administrators or test development
staff on the appropriate workstations. Alternatively all or some of these files could be transmitted from the central
processing site to the workstations electronically.
[O29O] The hard disk of each workstation is preferably configured during installation with at least one test program
3o directoryand an administration directoy. Thefiles loaded ontothe hard disk in thetest program directoy were described
in Section Ill with reference to Figures 29 and 31 . Figure 52 depicts the administrative files which are loaded into each
of these directories during installation.
[O291] Afile fortrackingthe histoy of activity on the center's workstations is created at installation in the administrative
directory and updated throughout each day the workstation is in use. It will be referred to as PCDATA 322. It is typically
35 a binary formatted file. It includes a workstation number 351 , a session number 352, a sequence number 353, a histoy
block 354, and spiralling information counters 355.
[O292] The workstation number 351 in PCDATA 322 is a preassigned number given to each workstation, such that
each workstation at one test center will have a unique workstation number. The session number 352, starts at zero
when PCDATA 322 is created at the time of the installation and is incremented each time the workstation undergoes
4o a cold start. The sequence number is also started at zero and is incremented each time a log record is written to the
hard disk. The sequence number is reset each time the session number 352 is incremented. The history block 354
contains information about the last N times the workstation was started, where N can be any integer. Preferably, the
histoy block 354 contains the workstation number 351 , session number 352, with a corresponding date/time stamp
and the name of the administrator who logged onto the workstation during that session, for each of the previous N
45 sessions. The spiralling information counter 355 is a value used to select test components for a particular CBT when
automatic selection, or spiralling, is to be used.
[O293] A station configuration file 324 is created at installation in the administrative directory. The stateion configu-
ration file is typically a binay formatted file. Preferably, the station configuration file includes the data elements de-
scribed below.
5O (PICTURE)
55 31
EP O 664 O41 B1
(PICTURE)
(continued)
5
1O [O294] The security log file 326 shown in Figure 52 is not created or installed during system installation 3OO. A
15 descr__pt__on of the secur__ty _og f___e 326 w____ therefore, be deferred.
[O295] Workstation start-up 3O2 shown in Figure 51 occurs when the computer's power is turned on or if someone
reboots the computer (i.e., it is well known in DOS to hit CTRL+ALT+DEL). After performing its own software checks
and loading procedures, collectively well known as BlOS routines, the computer loads DOS. After completing these
tasks, the computer preferably loads and executes a CBT security application. An example of such security software
2O __s SCuA, commerc__a__y ava___ab_e from _ach __ software.
[O296] The security phase 3O4 performs vital system checks of its own. In a preferred embodiment, these checks
would include an integrity check and a virus scan. Commercially available software is also well known for providing
these functions. If either check fails an error message may be provided on the computer's display monitor and the
workstation is rendered inoperable to any user without an authorization code. If no errors are detected, i.e., no file
25 tamper__ng or v__ruses, the secur__ty app___cat__on conf__gures for __guest mode__ operat__on. _n __ts __guest mode__, anyone can
use the workstation normally, except the security application will block access to the test program and administrative
directories.
[O297] The security software also provides three violation counters which are incremented during the security phase
3O4. The first counter records the number of times an unauthorized person attempts to change directories or drives to
3O any of the d__rector__es or dr__ves protected by the secur__ty software. The second v__o_at__on counter records the number
of partially successful matches to the secret administration code for initiating the administrative application which will
be described in detail below. A third violation counter records any attempts made to access the CBT data from low-
Ievel BlOS commands during the workstation start-up procedure 3O2.
[O298] Before beginning the administration application initialization procedure 3O6, the test administrator preferably
35 _ogs on to the CBT system. The adm__n__strator may be requ__red to key __n the secret adm__n__strat__on code wh__ch __s stored
on the hard disk or server during system installation. In a preferred embodiment, the security software will request the
administrator to also enter a password if the secret administration code was correctly keyed in. Then if both the secret
administration code and password, if requested, were entered correctly, the security software starts the administration
application initialization procedure 3O6.
4O [o2gg] Each test center __s preferab_y supp___ed w__th three types of d__skettes. Two Center un__que D__sks (CuD) are
provided to each test center for initiating the administrative application and for providing other necessary files for the
operation of the CBT system. A set of data disks are also provided to each test center periodically for storing the
examinee performance files, security log files and system error log files which are created by the test delivery and
administrative systems and stored on the hard disk or server. Additionally, backup disks are provided to each test
45 center for back__ng up data accumu_ated over a predeterm__ned per__od of t__me. The CuD __s used __n _n__t__a___zat__on Adm__n_
istration Application Procedure 3O6, and the data disks and backup disks are used by the Close-of-Day Procedure 31 O.
[O3OO] The first step of the Administration Application Initialization Procedure 3O6 is to interface with the security
software. In particular, the administrative application checks security software to determine if it is the proper version.
The administrative application then enables or disables appropriate workstation resources such as drives or printers
5O accord__ng the CBT f___es, and obta__ns the v__o_at__on counts from the secur__ty software. After ver__fy__ng that the proper
version of security software had been installed, the administrative application displays a message to the administrator
to insert the center unique disk.
[O3O1] The center unique disk contains three types of files; the key file, the logon file and the security file. The key
file contains a unique code assigned to the test center and the test center name. The logon file contains the adminis-
55 trat__ve app___cat__on _ogon _ D, password, author__ty _eve_, and the names of each person at the test center author__zed to
use the CBT system. Authority levels are associated with menu options of the administrative application; preferably
no administrator can execute options that require higher authority levels than that assigned his/her login l D in the logon
32
EP O 664 O41 B1
file. The security file contains some portion of code, such as a Dynamic Link Library, that is used by the Test Delivery
system. Optimally, the test delivery application cannot be started without the information in the security file on the
Center Unique Disk, which prevents unauthorized access to the delivery system.
[O3O2] When the administrator correctly inserts the center unique disk, the key file and security file are copied from
5 the center unique disk to the workstation's hard disk. Assuming these files are successfully copied, the administrator
is then prompted to logon bythe administration system 1 4. Then the administrator may enter his or her ID and password
which should match the information contained in the logon file. To protect this logon information from being accessed
on the center unique disk, the logon file may be hidden and encrypted as is well known.
[O3O3] Once the administrator correctly enters his or her l D and password, the Administration Application Initialization
1O Procedure 3O6 displays a main menu. The menu provides the administrator with at least three choices; administer test
procedure 3O8, initiate close-of-day procedure 31 O, or cancel which returns the administrator to the logon screen.
[O3O4] When the administrator selects the Administer a Test Procedure 3O8, a menu of tests available on the work-
station is displayed. The list is provided by the test program file 32O as shown in Fig. 52. The administrator then selects
the desired test to be delivered, and provides examinee information, such as name and registration number. The
15 administration system 1 4 then checks the registration number for conflicts. Conflicts can occur, for example, if the
registration number was already entered at the same workstation on the same day or if an Examinee Performance File
having the same registration number exists on the workstation's hard disk. If there are no conflicts, atest staging screen
is preferably displayed with the Examinee's name and registration number. When the Examinee arrives to take the
test, the administrator then enters a code to bring up an examinee confirmation screen.
2O [O3O5] The examinee confirmation screen presents the same and test information to the examinee. If the examinee
confirms that the information is correct, a record indicating that a test was administered is written to a security log file
and the test begins as provided by the test delivey system.
[O3O6] If changes to the information are necessary because the information is incorrect, the administrator can enter
the special key sequence to bring up an edit screen. Two options are available on this screen, ''Proceed'' and ''Termi-
25 nate''. lf the administrator selects ''Terminate'' the administrative application returns to the main menu. lf the adminis-
trator selects ''Proceed'' he/she must enter a valid logon and password and is then presented with an edit screen.
Preferably, any of the fields can be changed and a new test may be selected from the edit screen. For example, if the
examinee's registration number is incorrect, the administrator can indicate that the number is incorrect and will be
prompted to reenter it. The administrative confirmation screen then appears and processing continues as described
3O above.
[O3O7] The close-the-day procedure 31 O in Fig. 51 performs the necessary data transfer, backup and cleanup oper-
ations to shut a workstation down for the day. Referring now to Figure 53, the close-the-day procedure first closes out
the Security Log on the workstation's hard disk at 31 2. The administrative application checks whether tests were ad-
ministered during the current session at 31 4. If no tests were administered, a message is displayed at 31 6 indicating
35 that no copying is required. If tests were administered during the current session, the administrative application then
executes the Data Disk Procedure at 31 8.
[O3O8] The Data Disk Procedure 31 8 first prompts for insertion of a Data Disk. It then verifies that the inserted diskette
is a Data Disk. If an incorrect diskette is inserted, the administrator is again prompted for the Data Disk. This loop
continues until the correct diskette is inserted orthe workstation is powered down. When the correct diskette is inserted,
4O the security log file, system error log file (if any) and any examinee performance files are copied to the Data Disk. The
administrative application preferably checks whether the current Data Disk has enough space for each file before the
copy is attempted. If adequate space is not available, the administrator is prompted to insert another Data Disk.
[O3O9] The administrative application then executes the Backup Disk procedure 33O. It first prompts for insertion of
a Backup Disk. It then verifies that the inserted diskette is a Backup Disk. If an incorrect diskette is inserted, the
45 administrator is again prompted for the Backup Disk. Just like the Data Disk procedure described above, this loop
continues until the correct diskette is inserted orthe workstation is powered down. When the correct diskette is inserted,
the security log file, system error log file (if any) and any examinee performance files are copied to the Backup Disk.
Again, the administrative application preferably checks whether the current Backup Disk has enough space for each
file before the copy is attempted. If adequate space is not available, the administrator is prompted to insert another
5O Backup Disk.
[O31 O] After the files are copied, a message is displayed at 332 asking the administrator to remove the Backup Disk.
Processing, preferably, will not proceed until the administrator removes the disk. The administrative application then
deletes substantially all transferred files from the hard disk of the workstation at step 334. The administrative application
displays a message informing the administratorthatthe workstation was successfully closed and can be safely powered
55 down when the DOS prompt appears. To return to DOS, the administrator can, for instance, click on ''OK'' to dismiss
the message and exit the administrative application orthe administrative application may be exited automatically. When
the administrative application terminates, the administrator is returned to the ''guest mode'' and the DOS prompt ap-
pears. 33
EP O 664 O41 B1
[O311] It should be understood that in addition to the checks made by the administration system described above, it
would be well known to provide numerous other checks. Furthermore, any check made by the administration system
1 4 with an unfavorable result can cause the administration system to terminate and return the workstation to the guest
mode. Likewise, it would be well known to provide the administrator with cancel options to return the administrator to
5 the previous procedure or to the guest mode.
B. Se_urity Log File
[O312] As stated above, the security log file 326 shown in Figure 52 is not created or installed during system instal-
1O lation. Preferably, the security log file is created by the administrative system automatically each day and stored on
the workstation's hard disk in the administrative directory. Major system events such as security violations detected by
the security software, system start up/restart, rejected start up attempts, and initiation of main menu options (e.g.
administer a test, close-of-day, or restart a test) are recorded in log records which form the security log file. In order to
keep accurate log records of the initiation of system events, administrators preferably log in before executing any
15 administrative menu options or performing other activities that result in log records being written to the security log.
[O313] Referring to Figure 54, one example of the structure of a log record is shown. Generally, each log record has
a standard header 4OO followed by variable data 41 O. Specifically, the header may contain the following fields_. an
EVENT code 4O1 , DATA LENGTH 4O2, SEQUENCE NUMBER 4O3, CHECKSUM 4O4, ADMl NISTRATOR 4O5 and a
Tl MESTAMP 4O6. The EVENT CODE field 4O1 may contain a predetermined code assigned to each event to be
2O recorded in a log file. _able 1 O below provides some examples of events which may be recorded by the administrative
system in the security log file.
(PICTURE)
TABLE 1 O
25
3O
35
4O
45
5O
55 34
EP O 664 O41 B1
(PICTURE)
TABLE 1 O (continued)
5
1O
15
2O [O314] The DATA LENGTH field 24O2 provides the length of the data in bytes presented in the variable data portion
25 241 O of the log record. The SEQUENCE NUMBER 24O3 begins at 1 and increments each time a log record is written
to the hard disk. If the last sequence number does not correspond to a STOP log event, then the security log file may
be considered corrupt. A checksum may performed on the data in the log record as is well known and the result stored
in the CHECKSUM field 24O4. The name of the test center administrator who initiated the event is written to the AD-
MINISTRATOR field 24O5. The Tl MESTAMP field 24O6 records the time of day that the particular log record was added
3o to the security log file.
[O315] The variable data portion 41 O of the log record has one or more subfields depending on the event recorded
in the log record. Table 1 O above additionally lists some subfields recorded for those events listed in the Table.
C. Administrative Appli_ation Overview
35 [O316] Referring now to Figures 87 to 1 O1 F, a detailed description of the software will be described for implementing
the administrative application functions as described above.
[O317] The administrative application functions are separated into states by the software according to the present
invention. Functions are executed by respective state procedures. Each state procedure can preferably process two
4o actions_. Execute and End. Execute causes the state procedure to perform the functions associated with the state. End
causes the state procedure to perform implementation-dependent activities such as freeing memory or resources or
deleting temporary working files prior to transitioning to a new state.
[O318] In a preferred embodiment, the states defined for the administrative application and a brief description of each
are as follows_.
45 NULL STATE
The Administrative Application is initialized to this state when starting.
ADMIN STATE
This state is in effect during menu processing, the performance of menu functions not associated with system
5o maintenance (MAl NT state), close of day (CLOSE state), and the delivey of tests (TDS state).
MAINT STATE
This state is set when the administrator responds affirmatively to the ''Do you want to perform maintenance?''
query, which is displayed during start up in the NULL state. The state remains in effect, possibly for multiple main-
tenance updates, until the administrator indicates that maintenance is complete, at which time the system transits
55 to the NULL state.
CLOSE STATE
This state is set when the administrator selects Close-of-Day from the menu. It remains in effect until close of day
operations are complete, at which time the system transits to the EXIT state.
35
EP O 664 O41 B1
TDS STATE
The state is set from the ADMl N state when the Test Delivery Application is to be executed. It remains in effect
until the testing session is complete, at which time the system transits to the ADMl N state.
EXIT STATE
5 This state is set from the ADMIN state, when the administrator selects the Exit menu option, or from the CLOSE
state when close of day is complete. It remains in effect until the termination message is acknowledged or expires,
at which time the system transitions to the EXITl NG state.
EXITING STATE
This state is set from the EXIT state. It remains in effect only as long as it takes the system to exit to the operating
1O system.
[O319] The flow of processing in the Administrative Application begins in the Main-Procedure. A flow diagram of the
Main-Procedure is shown in Figure 55. Referring to Figure 55, the security software may be invoked at 1 OO3, if it is
determined by reading a flag in the station configuration file at 1 OO2 that a security application is to be used. Then the
15 Start-System-Procedure 1 O21 is invoked at 1 OO4. The Start-System-Procedure 1 O21 will be described below in con-
junction with Figures 56A and 55B. Generally, however, the Start-System-Procedure 1 O21 writes the start session
record, updates the history file, and performs password processing. When the Start-System-Procedure is completed,
the program returns to the Main-Procedure at 1 OO5 invoking the next state procedure corresponding to the current
state (i.e., Action is a variable, which defines the action to be taken next and here is set to execute).
2O [O32O] After executing the current state procedure, the program returns to the Main-Procedure at 1 OO6. Then it
checks return codes or other implementation-dependent mechanisms to determine whether the state procedure has
indicated that the state is to end at 1 OO6. If the current state is to end, the state procedure for the current state is
executed by setting Action -- End. The return value is the next state retrieved from the current state procedure and it
is returned by executing the current state procedure at 1 OO7. If the new state is the EXITl NG state execute the
25 Stop-System-Procedure at 1 OO9 or execute the current state procedure at 1 OO5. After the Stop-System-Procedure
1 O21 has been executed the application is exited to the operating system at 1 O1 O.
1 . Start-System-Pro_edure
3O [O321] Figures 56A and 56B provide a flow diagram of the Start-System-Procedure. If the station is diskless, check
whether the system state is CLOSE, XMIT, or MAINT as at 1 O23, starting the station will interrupt the close or mainte-
nance operation that is in progress so an error message at 1 O24 is preferably displayed and the procedure exited to
the operating system at 1 O27. Otherwise, the station has a floppy disk, so a check may be made to determine whether
the master station is already operating at 1 O22. If so, this station does not need to be started, so an error message is
35 preferably displayed at 1 O24 and the procedure is exited to the operating system at 1 O27.
[O322] Load the station configuration file. Check the element that declares whether the Center Unique Disk (CUD)
is to be used, and whether this station is diskless. If so, display informative message requesting the start of the master
station. Loop displaying the message until the master station is started. If the user cancels, exit to the operating system.
[O323] If the CUD is to be used, and this workstation is equipped with afloppy drive as shown at 1 O28, an instructional
4O message requesting the administrator insert the CUD is displayed at 1 O29 until a valid CUD is inserted. If the admin-
istrator cancels, a Start Rejected record is written to the Security Log File at 1 O3O (reason -- Canceled) and the program
exits to the operating system.
[O324] If the workstation is a stand-alone system, copy the KEY file, LOGON file, and SECURITY file from the CUD
disk to the station's hard disk. If the workstation is in network system, copy the SECURITY file to the server.
45 [O325] After the workstation has been properly configured according to Figure 56A, the flow diagram of the
Start-System-Procedure 1 O31 continues in Figure 56B. Referring to Figure 56B, the list of installed testing programs
(see figure 52) is automatically loaded onto the workstation's hard disk at 1 O32. Then the Login-Procedure, which will
be described below, may be executed at 1 O33. If the administrator cancels the Login-Procedure, a Start Rejected
(reason -- Canceled) record is written at 1 O37 to the Security Log file and the application exits to the operating system.
5O If the user fails to login within a specified number of tries, a Start Rejected (reason -- Login Failure) record is written
at 1 O37 to the Security Log file and the application exits to the operating system.
[O326] Next the element of the station configuration file that declares whether workstation numbers are to be used
is checked at 1 O34. If workstation numbers are to be used, a screen is displayed requesting the entry of a valid station
number at 1 O35. One example of such a screen is shown in Figure 57. As shown in Figure 57, it is preferable to permit
55 the administrator to cancel at this point in the test administration process. lf the administrator cancels, a Start Rejected
record is written to the Security Log file (reason -- Canceled) at 1 O37 and the application exits to the operating system.
Alternatively, if the administrator has input a workstation number at 1 O35, the current station number is set to the
number entered at 1 O35. 36
EP O 664 O41 B1
[O327] The Process-State-Procedure may then be executed at 1 O36. Although the Process-Start-Procedure will
be described in more detail, it is generally responsible for checking on restart conditions and for allowing the admin-
istrator to perform maintenance functions.
5 2. Login-Pro_edure
[O328] The Login-Procedure 1 O4O will now be described with reference to Figure 58 which provides a flowchart of
the steps executed by the Login-Procedure software. First a Login screen is preferably displayed on the workstation
prompting the administrator to login with his or her Login l D and enter his or her password. An example of the Login
1O screen is shown in Figure 59. The login l D and password entered by the administrator are accepted at 1 O41 . The login
file is then searched at 1 O42 for a record matching the Login l D entered by the administrator at 1 O41 . If no match is
found, an error message is preferably displayed at 1 O43 and steps 1 O41 and 1 O42 are repeated until a caller-specified
number of attempts is exhausted. If the specified number of attempts to login have been made without finding a Login
l D match, the Login-Procedure 1 O4O is preferably exited with an indication of Login error.
15 [O329] Upon successful login at 1 O42, the password entered by administrator is compared at 1 O44 with one or more
passwords stored in the login record. If a match exists, program control is returned to the Start-System-procedure
indicating that the administrator has successfully logged on. However, if there is no password match at 1 O44, a default
password is constructed at 1 O45. The default password is constructed by an algorithm in the software. The same
algorithm may be used by predesignated Support Testing staff or other staff to create a new password that the system
2O will recognize for an administrator who calls with a forgotten password. The default password is then compared at 1 O46
to the password entered by the administrator. If a match is found, program control is returned to the
Start-System-Procedure indicating that the administrator has successfully logged on with the default password.
3. Pro_ess-State-Pro_edure
25 [O33O] A flow diagram of the Process-State-Procedure 1 O5O is shown in Figure 4. At 1 O51 , a check is made to
determine whether testing sessions are available for restart. These are sessions that ended abnormally due to, for
instance, system failures; they are identified because the station status in the session status table is TDS Active (see
State-TDS-Procedure, below). An informative message is preferably displayed indicating that testing sessions are
3O available for restart, if any, at 1 O52.
[O331] The current system state is checked at 1 O53 and a message informing the administrator whether the current
state is CLOSE is displayed at 1 O54. In this state, the close operation is preferably completed before other activities
can be initiated on the workstation.
[O332] If a CUD was inserted, an instructional message is displayed at 1 O55 requesting that the administrator remove
35 the diskette and store it properly. Preferably, the Process-State-Procedure 1 O5O will loop until the diskette is removed
so that the CUD cannot be inadvertently or intentionally tampered with or accessed by unauthorized persons.
[O333] The current state is checked at 1 O56 to determine if it is the NULL state. If the current state is the NULL state,
the administrator is queried to perform maintenance at 1 O59. However, if the workstations at the test center are net-
worked, then it is preferable that only the master station be provided with the capability of invoking the maintenance
4O procedure. Therefore, checks may be made at 1 O57 to determine whether or not the workstation is a stand alone
system or part of a network of workstations and if it is networked a check is made at 1 O58 to determine whether the
workstation is a master station. If maintenance is to be performed as indicated by the administrator at 1 O59, the current
state is set to MAINT at 1 O6O.
[O334] If the current state was not NULL at 1 O56 or the administrator did not wish to perform maintenance at 1 O59,
45 program control is returned to the Start-System-Procedure 1 O21 . Additionally, if the workstation is networked and it
is not a master station, it is preferable to return to the Start-System-Procedure 1 O21 without providing an opportunity
for the administrator to perform maintenance.
4. State Pro_edures
5O a. Null-State-Pro_edure
[O335] Aflowchart of the State-Null-Procedure 1 O7O is shown in Figure 61 . Before executing this procedure, a check
is preferably made at 1 O71 to determine if Action has been set to End. However, assuming Action has not been set to
55 End at 1 O71 , a Start Session record is written at 1 O73 to the Security Log file. The administrator is then prompted at
1 O74 as to whether he or she wishes to change the password. If the workstation is a stand alone station, and station
numbers are in use, and this is station number 1 , which is preferably the only station from which passwords can be
changed in a center without a local area network in order to allow administrators to change passwords only once for
37
EP O 664 O41 B1
every station in the center, display a query asking the administrator if he or she wants to change passwords. If the
response is affirmative, the Change-Password-Procedure is invoked at 1 O75. The Change-Password-Procedure will
be described in detail below.
[O336] If the workstation is a stand alone station, or the workstation is a master workstation in a networked system,
5 then the PC Histoy in the PCDATA file is updated at 1 O76. Then Action is set to End at 1 O77. Thus when the program
returns to the Main-Procedure at 1 OO6 in Figure 1 , the state is checked to determine whether Action is set to End at
1 O76 and the Null-State-Procedure is called again at 1 O77.
[O337] Referring back to Figure 61 , since the action state is set to End at step 1 O71 of the Null-State-Procedure,
the next state procedure is set to ADMl N at 1 O72. Then the program returns to the Main-Procedure with the current
1O state set to ADMlN.
i. Change-Password-Pro_edure
[O338] If the administrator indicates that he or she desires to change his or her password, the State-Null-Procedure
15 invokes the Change-Password-Procedure as described above. A flowcha_ of the Change-Password-Procedure is
shown in Figure 62. Before the password is changed it is preferable to save a copy of the current login file as shown
at 1 294. The workstation's floppy drive may then be enabled at 1 295.
[O339] A screen requesting the administrator whose password is to be changed to login is displayed at 1 296 and the
Login-Procedure is called. If the administrator successfully logs in, a screen requesting entry of the new password is
2O displayed at 1 298 until the administrator enters the new password. ln a preferred embodiment, a screen requesting a
second entry of the new password is displayed at 1 298 until the administrator enters the new password. In this preferred
embodiment, each enty is stored separately as first and second password entries.
[O34O] Then the first and second password entries are preferably compared at 1 3OO. If they do not match, an error
message may be displayed as shown at 1 3O2 and the administrator may be given an opportunity to login again at
25 1 296. lf the password entries match, the login record for this administrator is updated in the login file at 1 3O4 with the
new password. In preferred embodiments, a flag is set to indicate a login record has been changed.
[O341] The administrator maythen be queried at 1 3O6 by displaying a message asking whether there are more logins
records to change. If the administrator indicates he or she wishes to change more passwords, then the next adminis-
trator whose password is to be changed logs in at 1 296.
3O [O342] When the administrator does not wish to change any more passwords and at least one login has been
changed, an instructional message requesting the administrator to insert the CUD is preferably displayed at 1 31 2 until
the CUD is inserted. The login file stored on the CUD may then be overwritten with the login file written to the work-
station's local memory at 1 31 4. An instructional message requesting the administrator to remove the CUD is preferably
displayed until the CUD is removed bythe administrator at 1 31 4. If none of the logins have been changed as determined
35 at 1 31 O, access to the workstation's floppy drive is disabled at 1 3O8. The program then returns to the caller.
[O343] In a preferred embodiment, the administrator should be permitted to avoid overwriting the login file even when
changes have been made by cancelling the procedure. If the administrator cancels, the CUD need not be inserted,
but rather the floppy drive is disabled at 1 3O8 and the program returns to the caller.
4O b. __a_e-Ad_in-Pro_edure.
[O344] A flow chart of the State-Admin-Procedure is shown in Figure 63. When the procedure is first invoked Action
is set to Execute. Therefore, when the state is checked at 1 O81 , the state is not set to End and the procedure continues
with step 1 O83. The administrator is requested to login according to Perform the Login-Procedure. As noted above,
45 the administrator preferably logs in again before initiating any new activity so that the identity of the initiator can be
Iogged accurately in the Security Log File.
[O345] The date and time of the last update of the administrator's password is checked at 1 O85. If the password is
out of date as defined by the PASSWORD-Ll MIT element of the station configuration file, the password can be updated
using the Change-Password -Procedure. In a preferred embodiment, a screen will be displayed at 1 O84 for the ad-
5O ministrator permitting the administrator to change his or her password.
[O346] A timer may then be started and the system or main menu is displayed at 1 O86. Figure F3 shows a screen
displaying the main menu in a preferred embodiment. Preferably, the authority level required for each option is checked
against the administrator's authority level as defined in the login file while displaying the menu. Then, if the current
administrator does not have sufficient authority for a menu option, the option is preferably disabled.
55 [O347] The administrator then selects a menu option as shown at 1 O87. The procedure address associated with the
menu option is retrieved and executed at 1 O88. In a preferred embodiment, the supported menu options and associated
procedures include_. 38
EP O 664 O41 B1
A- dminister Operational Test
(Menu-OpTest-Procedure)
5 A- dminister Demonstration Test
(Menu-DemoTest-Procedure)
-Restart a Test
(Menu-RestartTest-Procedure)
1O -
Close Day
(Menu-CloseDay-Procedure)
-Exit
15 (Menu-Exit-Procedure)
-Logon Maintenance
(Menu-LogonMaint-Procedure)
2O -Change Password
(Menu-ChgPassword-Procedure)
A- bout
(Menu-About-Procedure)
25 Each of the above listed Menu Procedures will be described in detail below.
[O348] If after the selected menu option is executed, the procedure indicates a new state is required at 1 O89, the
program returns to the Main-Procedure and the State-Admin-Procedure state is set to end. If a new state is not
indicated at 1 O89, the administrator is prompted to login at 1 O83. The flowchart forthe Main-Procedure shown in Figure
3O 1 at steps 1 OO5 and 1 OO6, indicates that the State-Admin-Procedure will be called again. If a new state had been
returned by the menu procedure, the State-Admin-Procedure would indicate that the state was set to end at step
1 O81 . The next state returned by the menu procedure executed at 1 O88 would then be returned to the Main-Procedure.
[O349] If the Inactivity Timer as defined in the station configuration file expires, the menu display is reset to the login
screen at 1 O9O. This prevents an unauthorized individual from starting a menu option, should the administrator be
35 interrupted while initiating an activity.
_. State-Close-Pro_edure.
[O35O] A flowchart of the State-Close-Procedure is shown in Figures 65A and 65B. When the procedure is first
4O invoked Action is set to Execute. Therefore, when the state is checked at 1 O81 , the state is not set to End and the
procedure continues with step at 1 1 O3.
[O351] Preferably, the procedure checks whether the workstation is equipped with a floppy drive at 11 O3. If it is not,
an error message is preferably displayed; the next state is set to ADMl N and the State-Close-Procedure is set to End
at 1 1 O4.
45 [O352] If the workstation is part of a networked system and other stations are active as determined at 11 O5, then an
error message is preferably displayed at 1 1 O4.
[O353] A check is made at 1 1 O6 to determine whether the workstation has been restarted. If not, a reconciliation
process occurs. The hard disk of the station in standalone systems or of the server in networked system is scanned
at 1 1 O7 to count the number of sessions that have been administered for each testing program.
5O [O354] The list of testing programs and the corresponding count for each of the administered tests are then preferably
displayed at 1 1 O7. In preferred embodiments, the list will contain space for the administrator to enter a count as derived
from paper logs as shown at 11 O8.
[O355] Another list containing the name of each testing program, the counts generated automatically by the system
and the counts generated manually by the administrator for each, and a place to indicate whether a paper report will
55 be submitted may then be displayed. lf the system and administrator counts of any element in the list differ, it is pref-
erable to permit the administrator to change the manually prepared count and/or enter a note explaining the discrep-
ancy.
[O356] A list of all examinee results, performance files, security log files, and system error files, which are to be
39
EP O 664 O41 B1
returned for processing may then be generated at 11 1 O. The system prepares this list by scanning the hard disk of the
workstation in standalone environments or of the server in networked environments.
[O357] All of the files in the list produced in step 1 1 1 O may then be copied to the Data Disk(s) at 1 1 1 1 . Preferably, an
indication that this step has been performed is made, i.e., set an appropriate flag in software, so that in the event the
5 workstation is restarted after the offload has been performed the offload procedure is not necessarily repeated. Fol-
Iowing the offload of files to the data disks at 11 1 1 , it is preferable to also copy all of the files in the list produced in step
1 1 1 O to the Backup Data Disk(s). Again it is preferable to provide some indication that this back up step has been
performed.
[O358] All files named in the list produced in step 11 1 O should then be preferably erased from the workstation's hard
1O disk at 1 1 1 2. Providing an indication that the erasure has been performed is also preferable.
[O359] The next state should then be set to the EXIT state and the state of the State-Close-Procedure is set to End
at 1 1 1 3. The program then returns to the Main-Procedure, which will then reexecute the State-Close-Procedure to
get the next state. Upon reexecuting the State-Close-Procedure, it will be determined at 11 O1 that the state has been
set to End and the next state, i.e., EXIT, will be returned to the Main-Procedure from 11 O2.
15 [O36O] In a preferred embodiment, the administrator is permitted to cancel the close-of-day procedure. If the admin-
istrator cancels, the next state should be set to ADMl N. Therefore, the next state retrieved at 11 O2 may be one of
ADMl N or EXIT.
d. State-Maint-Pro_edure
2O [O361] A flowchart of the State-Maint-Procedure is shown in Figure 67. Preferably, the procedure checks to deter-
mine whetherthe workstation has been restarted at 11 23. If it has been restarted, the administrator is preferably prompt-
ed to determine whether he or she wishes to perform more maintenance at 11 24. If no more maintenance is to be
performed, the next state is set to ADMIN at 11 25, and the program is returned to the Main-Procedure. If more main-
25 tenance is to be performed at 1 1 24, the maintenance program is executed at 1 1 3O. A variety of maintenance programs
could be used, such as that commercially available from Microsoft Corporation as part of the Windows Software De-
velopment Kit.
[O362] If the workstation has not been restarted as determined at 11 23, the procedure preferably checks at 11 26
whether any testing sessions have been performed since the last CLOSE state was executed. If so, an error message
3O is preferably displayed and the next state is set to the ADMl N state at 11 29.
[O363] If the workstation is part of a network as determined at 11 27, the procedure then may check whether any
workstations are still active at 1 1 28. If there are active workstations in the network, an informational message is pref-
erably displayed and the next state is set to ADMl N at 11 29. This prevents that administrator from performing mainte-
nance while a testing session is in progress.
35 [O364] If the workstation is not part of a network at 11 27 or if there are no active workstations in the network at 11 28,
the maintenance program can then be executed at 11 3O. The state of the procedure is then set to End.
[O365] If the state of the State-Maint-Procedure is set to End as determined at step 11 21 , the list of installed testing
programs are reloaded from the hard disk of the standalone station or server at 11 22 and updated if testing programs
have been added or removed. A record is then preferably written at 11 31 to the Security Log File indicating that main-
4O tenance has been performed. In a preferred embodiment, the next state is set to the NULL state at 11 32 and the
program is returned to the Main-Procedure.
[O366] Preferably the administrator is permitted to cancel the maintenance procedure. If the administrator cancels,
the next state should be set to ADMl N and the state of the State-Maint-Procedure is set to End. The program may
then be returned to the Main-Procedure.
45 e. State-TDS-Pro_edure
[O367] A flowchart of the State-TDS-Procedure is shown in Figure 68. When the procedure is first invoked Action
is set to Execute. Therefore, when the state is checked at 11 51 , the state is not set to End and the procedure continues
5O with step 1 1 54.
[O368] The administrative application prompts the administrator to enter information about the examinee and creates
a System Parameter file containing this information at 11 54. Preferably, the Start Session Record of the examinee
performance file is appended to the System Parameter file.
[O369] The Test Delivery Application may then be executed. An indication that the testing session is open and active
55 should then be provided at 11 55, e.g. a TDS active flag may be set in a session status table maintained by the admin-
istrative application. When the testing session is complete, the Test Delivery Application will return program control to
the State-TDS-Procedure at 11 56. The next state is then set to ADMl N at 1 1 56 and the state of the
State-TDS-Procedure is set to End at 1 1 57. 4O
EP O 664 O41 B1
[O37O] The program then returns to the Main-Procedure, which reexecutes State-TDS-Procedure with the state set
to End. The TDS Active flag in the status table for the testing session can then be updated with returned status at 11 52.
An informational message indicating that the testing session is complete is preferably displayed at 11 53 and the program
returns to the Main-Procedure with the next state set to ADMl N.
5 f. State-E_it-Pro_edure
[O371] A flowchart of the State-Exit-Procedure is shown in Figure 7O at 1 1 6O. An exiting informational message is
preferably displayed at 1 1 62 when this procedure is executed. In a preferred embodiment, the exiting message is
1O displayed until the administrator acknowledges the message or a predetermined time limit expires as shown at 11 63.
[O372] The next state is set to EXITl NG at 1 1 64 and the program returns to the Main-Procedure to invoke the stop
procedure.
5. MENU PROCEDURES
15 a. Menu-OpTest-Pro_edure
[O373] When the ''Administer Operational Test'' is selected from the main menu the State-Admin-Procedure invokes
the Menu-OpTest-Procedure. A flow diagram of the Menu-OpTest-Procedure is shown in Figure 71 . The
2O Menu-OpTest-Procedure at 1 1 7O sets a test flag to indicate delivery of an operational test at 1 1 71 and calls the
Menu-TestCommon-Procedure at 1 1 72. The Menu-TestCommon-Procedure will be described in detail below.
b. Menu-DemoTest-Pro_edure
25 [O374] When the ''Administer Demonstration Test'' is selected from the main menu the State-Admin-Procedure in-
vokes the Menu-TestCommon-Procedure. A flow diagram of the Menu-DemoTest-Procedure is shown in Figure 72.
The Menu-DemoTest-Procedure sets a test flag to indicate delivey of demonstration test at 1 1 75 and calls the
Menu-TestCommon-Procedure at 1 1 76. The Menu-TestCommon-Procedure will be described in detail below.
3O _. Menu-TegtCommon-Pro_edure
[O375] A flowchart of the Menu-TestCommon-Procedure is shown in Figures 73A and 73B. A screen is displayed
at 1 1 8O listing substantially all of the installed testing programs. From that list, the administrator may select a testing
program at 1 1 81 . A screen listing the packages installed for the selected testing program is then preferably displayed
35 at 1 1 82. The administrator may then select the appropriate package from the list. A package contains all the information
needed to deliver a test, and a testing program may offer several tests, for example, Praxis Reading, Writing, and Math.
[O376] The selected package may then be checked at 1 1 83 to ensure that it includes an operational or demonstration
test as indicated by the test flag. If there is a mismatch, an error message is preferably displayed at 11 79.
[O377] If the selected package includes an operational test or demonstration test as indicated by the test flag, the
4O validation module for the selected package is loaded at 1 1 84. The validation module contains the edit and other rules
in effect for the testing program's examinee information. It also contains spiralling rules that control selection of test
components not selected bythe administrator, such as the random selection from among multiple scripts. The Examinee
Information screen is then displayed at 1 1 77. An example of an Examinee Information Screen is shown in Figure 74.
Preferably, the administrator is permitted to enter examinee related information, specify the type of candidate, and
45 select the type of test to be delivered. Additionally, the administrator preferably may enter an electronic note that will
be attached to the examinee's performance file. Such information may be entered at 1 1 85.
[O378] When the administrator indicates the Examinee Information screen is complete, the validation module is called
at 1 1 86 to validate the information. If the validation fails, an error message is preferably displayed at 11 73.
[O379] If the examinee information was entered merely to record a 'no show' as determined at 11 87, a 'No Show'
5O record is written to the Security _og File at 1 1 88 and return to the menu procedure from which the
Menu-TestCommon-Procedure was called.
[O38O] Turning now to Fig. 73, the Administrator's Confirmation screen is preferably displayed at 11 89 assuming the
Examinee has arrived to take the test. An example of the Administrator Confirmation screen is shown in Figure 75.
Substantially all key combinations are preferably locked out at 1 1 89 except for a secret administrator's override key
55 combination. The Administrator's Confirmation screen is displayed until the secret override key combination is entered
at 1 1 89.
[O381] The Examinee's Confirmation screen may then be displayed at 1 1 9O. An example of the Examinee Confir-
mation screen is shown in Figure 76. Preferably, the procedure will provide at least two options at this point, an override
41
EP O 664 O41 B1
from the administrator or a continue from the examinee as shown at 11 95.
[O382] If an override key combination from the Administrator is received, the administrator will be queried by a screen
such as that shown in Figure 77 whether he or she wishes to edit the examinee information or terminate the session.
If the administrator responds with 'terminate' at 11 96, the examinee information is preferably discarded and the program
5 returns to the caller. A Proceed or Terminate Screen will appear as shown in Fig. 77. If the administrator responds with
'edit' at 1 1 96, the Login-Procedure is invoked at 11 97. When the program returnstothe Menu-TestCommon-Procedure
from the Login-Procedure, the Examinee Information screen is again displayed at 11 77 (referring back to Fig. 74) to
permit editing of examinee information.
[O383] If the examinee responds with a 'Continue' at 11 95, a screen is preferably displayed at 1 1 91 through which
1O the examinee can enter his or her Identification Number and/or other biographical information which identifies the
examinee such as that shown in Fig. 76.
[O384] The spiraling procedures, which are program-specific rules-based procedures in the validation module, may
then be invoked at 1 1 92 to randomly select any test information not manually selected by the administrator.
[O385] The examinee information and test selection information is then preferably saved in this examinee's perform-
15 ance file at 1 1 93. The 'Administer Test' record may then be written to the Security Log File at 11 94. The validation
module may then be unloaded at 11 98. The state is preferably set to the TDS state at 11 99 and the program is returned
to the caller.
[O386] If the administrator cancels, which can occur at any point during this procedure when the administrator clicks
on the CANCEL button on the screen, the procedure returns to the caller.
2O d. Menu-RestartTest-Pro_edure
[O387] A flowchart of the Menu-RestartTest-Procedure is shown in Figure 78. When the Restart a Test option is
selected from the Main Menu, a restart test screen containing a list of installed testing programs may be displayed by
25 the Menu-Resta_Test-Procedure at 1 2O1 . An example of a restart test screen is shown in Figure 79. Each line in the
list contains both the name of the testing program and the number of restartable sessions found for each testing pro-
gram. The system identifies the restartable sessions by scanning its status tables and retrieving information from the
associated System Parameter files. The administrator selects one of the identified sessions to be restarted at 1 2O2.
Figure 8O shows an example of a screen displaying the available session for restart from which the administrator can
3O select.
[O388] The administrator may then choose a testing program at 1 2O3. The administrator may also preferably choose
to cancel. If the administrator cancels, the program returns to the caller. However, if the administrator has selected a
testing program at 1 2O3, the sessions available for the restart are preferably displayed. An example of such a screen
is shown in Figure 79. The administrator then preferably selects one of the listed sessions to restart at 1 2O4. The
35 administrative application then sets the session's status in the status table to indicate that it is now open. If in the
intervening time the session was selected for re-activation at a different station, so that the status now indicates it is
open, an error message is preferably displayed and continue at 1 2O1 .
[O389] When a session is reopened at 1 2O6, the Examinee Confirmation screen may then be displayed at 1 2O8 from
the information stored in the Session file. In a preferred embodiment, the examinee is prompted to continue from the
4O Examinee Confirmation screen displayed after restarting the session. When a signal to continue is received from the
examinee, the next state is set to the TDS state at 1 21 O. In a preferred embodiment, the session is cancelled if no
continue is received within a predetermined period of time, e.g. 6O sec., and the program returns to the caller.
e. Menu-CloseDay-Pro_edure.
45 [O39O] Upon selection of the Close Day option from the main menu, the Menu-CloseDay-Procedure at 1 21 3 sets
the next state is to the CLOSE state at 1 21 6 shown in the flowchart of Figure 81 and the program returns to the caller.
f. Menu-E_it-Pro_edure.
5O [O391] In standalone stations, the Exit option on the main menu is used to leave the administrative application without
creating data and backup disks. This option will only be available if the Exit option in the station configuration file is set
to Y. In LAN-based centers, the Exit menu option is used to leave evey station but the last. The Closeof-Day option
is then selected on the last station to shut the system down.
55 [O392] A flowchart of the Menu-Exit-Procedure is shown in Figure 82. After the Exit option has been selected by
the administrator from the main menu, the Menu-Exit-Procedure is invoked. This procedure may first determine at
1 226 whether the workstation is networked to other workstations or whether it is a stand alone workstation. If the
workstation is networked, the procedure then may check at 1 224 whether the workstation is the last active workstation
42
EP O 664 O41 B1
in the network. If it is the last active workstation in the network, i.e., the last to exit the administrative application, an
informational/query message is preferably displayed at 1 222 informing the administrator of that fact and seeking con-
firmation. If the administrator affirms the exit option at 1 222, the next state is set to the EXIT state at 1 232 and the
program returns to the caller.
5 [O393] If the administrator cancels the exit option from the informational/query message at 1 222, the program returns
directly to the caller without changing the state.
[O394] If it is determined at 1 224, that other workstations are active, a message indicating that there are other active
workstations is preferably displayed at 1 225. Then the next state may be set to the EXIT state and then the program
returns to the caller.
1O [O395] If it is determined at 1 226, that the workstation is a standalone system, the workstation's configuration is
checked at 1 228 to determine if Exit is permitted. Where Exit is not permitted, the program returns directly to the caller
without setting the next state to the EXIT state. However, when Exit is permitted, the procedure then may make a
determination at 1 23O whether any testing sessions have been administered since the last close of day. If one or more
tests have been administered since the last close of day, an informational/query message to that effect is preferably
15 displayed. lf the administrator affirms his/her desire to continue the Exit, the next state is set to the EXlT state at 1 232.
The program is then returned to the caller.
g. Menu-LogonMaint-Pro_edure
2O [O396] The Menu-LogonMaint-Procedure is called from the State-Admin-Procedure when the administrator selects
the Logon maintenance option from the main menu. A flowchart of the Menu-LogonMaint-Procedure is shown in
Figures 83A and 83B. If the security shell is installed, the workstation's floppy drive is preferably enabled at 1 242. The
procedure then may check whether the workstation is configured as a stand alone system or whether it is networked
with other workstations at 1 246.
25 [O397] If the workstation is a standalone system, but it is not designated as workstation number 1 , the preferred
station from which logon maintenance can be performed, an error message is preferably displayed at 1 25O and the
program returns to the caller. If the workstation is a stand alone system and it designated as workstation 1 , the logon
file may be saved at 1 252 for restoration in the event the function is canceled.
[O398] If the workstation is determined to be in a networked system at 1 246, the CANCEL function is preferably
3O disabled at 1 244. A screen containing the list of logon records in the logon file may then be displayed at 1 254. In a
preferred embodiment, the administrator should be permitted to add a new logon, as well as change or delete an
existing logon. The screen listing the logons preferably provides a means by which the administrator can indicate that
he or she has made all of the additions, changes, etc. that he or she desires. When the administrator indicates he or
she is finished with logon maintenance at 1 256, the administrator may preferably cancel the changes at 1 258, and
35 restore a local copy of login file from the back up copy. If the administrator does not cancel the changes at 1 258, a
'Logon Change' record is written to the security log file at 1 262.
[O399] In a preferred embodiment, when the workstation is configured as a stand alone system, an instructional
message requesting the administrator to insert the CUD is displayed at 1 266. After the CUD is properly inserted into
the workstation's floppy drive, the updated copy of the logon file is copied to the CUD at 1 268. After the logon file is
4O copied, an instructional message requesting the administrator to remove the CUD may be displayed at 1 266 until the
CUD is removed.
[O4OO] If the security shell had been installed, the floppy drive is preferably disabled at 1 27O and the program returns
to the caller.
45 _. Menu-__gPaggword-Pro_edure
[O4O1] The State-Admin-Procedure invokes the Menu-ChgPassword-Procedure when the administrator selects
the ''Change Password'' option from the main menu. If the workstation is a stand alone system as determined at 1 282,
an error message informing the administratorthatthis menu option is only supported on networked systems is preferably
5O displayed at 1 284 and the program returns to the caller. lf the workstation is networked to other workstations, a logon
informational screen is preferably displayed at 1 286. Requesting the administrator to enter his or her login l D and
password. The screen is preferably displayed at 1 286 until the login l D and current password are entered by the
administrator. If the administrator cancels, the program returns to the caller without effecting a password change.
[O4O2] After the logon ID and password have been entered at 1 286, a screen prompting for the new password may
55 displayed at 1 1 88. ln a preferred embodiment, the new password will be prompted for two times. The first and second
entries are then compared at 1 29O. If the two entries do not match, the administrator is preferably prompted two more
times to enter the new password.
[O4O3] If the two entries do match at 1 29O, the login file of the administrator is updated at 1 292 with the new password
43
EP O 664 O41 B1
and the program returns to the caller.
Menu-About-Pro_edure
5 [O4O4] Figure 86 provides attachment of the Menu-About-Procedure. As shown at 1 326, the
Menu-About-Procedure displays a screen containing an identifier and the installed version number of each of the
following_.
- Test Delivery System
1O - Each installed package for each installed testing program
[O4O5] It then returns to the caller.
5. Stop-System-Pro_edure.
15 [O4O6] When the next state has been set to EXITl NG, the Main-Procedure calls the Stop-System-Procedure. A
flowchart of the Stop-System-Procedure is shown in Figure 85. Any implementation dependent activities such as
removal of temporary working files at 1 321 , freeing of memory or other resources should be completed. The keyboard
and mouse filters are preferably de-installed at 1 322 priorto exiting to the workstation's operating system. The program
2O then returns to the Main-Procedure which exits to the workstation's operating system.
V. The Network Data Distribution System (NDDS)
A. Fun_tional Overview
25 [O4O7] The overall function of the Network Data Distribution System (NDDS) is to process data returned to the central
processing sites from the test centers and to distribute that data to the appropriate program specific production post-
processing systems e.g. GRE, SAT, etc. Typically current postprocessing systems are designed to process only one
record per examinee. Therefore, the NDDS serves as an interface between CBT and the various post processing
3O systems by transforming numerous examinee performance records into only a single file per examinee. The NDDS
functions are implemented by the NDDS application software. Preferably the NDDS software supports a multiuser
menu based system. Although, it should be understood that substantially any computer system could be used, the
NDDS application software is preferably run on a LAN located at the central processing site.
[O4O8] One objective of the NDDS is to process returned examinee data as generically as possible in order to ac-
35 commodate substantially all current and future test participants. In a preferred embodiment, the inputs and outputs of
the NDDS are shown in Figure 87. The examinee data processed by the NDDS 2OO2 is shown at 2OO1 and includes
examinee performance files, security log files, error log files, and demo files which will collectively be referred to as
transmission files. Additionally, a transmission header file is also preferably provided as an input 2OO1 to the NDDS.
Table 1 1 , below generally describes each of these input files.
4O (PICTURE)
TABLE 1 1
45
5O
55 [O4O9] The input files 2OO1 are then processed bythe NDDS 2OO2. The NDDS preferably consists of eight processing
44
EP O 664 O41 B1
components_. 1 ) file processing, 2) security/event log processing, 3) examinee performance record processing, 4) post/
format processing, 5) report processing, 6) essay file processing 7) reject/resolution processing and 8) CBT information
processing. A detailed description of these processes will be provided below.
[O41 O] The processed files may then be used by the NDDS 2OO2 to provide a number of different reports 2OO3 as
5 shown in Figure 1 . The NDDS 2OO2 further utilizes other information to process the input files 2OO1 . This information
is stored in NDDS files which may be categorized as either system files or application files. In preferred embodiments,
the system files will include those files listed in Table 1 2, below, and the application files will include those files listed
in Table 1 3, below.
1o (PICTURE)
TABLE 1 2
15
2O
25
3O (PICTURE)
TABLE 1 3
35
4O [O411] The NDDS generates both output files 2OO3, examples of which are listed in Table 1 4 below and report files
45 2OO4, liSted in T_ble 1 5 belOW.
(PICTURE)
TABLE 1 4
5O
55 45
EP O 664 O41 B1
(PICTURE)
Table 1 5
5
1O
15
2O These output files 2OO3 and report files 2OO4 contain data used to generate the reports 2OO5 shown in Figure 87. The
reports 2OO3 are used by a CBT test center operation group for operating and maintaining test centers.
[O412] A display of the NDDS Main Menu is shown in Figure 88. The Version l D that appears on this screen will be
taken from the Version l D file. The l D in that file is preferably updated each time a new release of the NDDS is installed.
25 The details of each of the NDDS processing components used to perform the menu options shown in Figure 88 are
described below.
1 . File Pro_essing Component
3o [O413] This component receives the transmission data from the test centers from various media. Preferably, the data
received is via one of two media. The first being 3.5 or 5.25 inch diskettes. Each diskette preferably contains a header
record. The second media is via modem to modem transmission. To better accommodate the NDDS network, all files
received from a test center, via data transmission, are preferably bundled in a single compressed file.
[O414] Therefore, the first file process is preferably to unbundle the transmitted data into their original file formats
35 for processing. A number of programs are commercially available to compress data and may be used. It is also pref-
erable that the product used to bundle or compress the data also provides a checksum facility to verify that the data
sent from the test center was received by the NDDS in the same format.
[O415] As mentioned above, there should be at least three files returned from the test centers - the Examinee Per-
formance Record (EPR) files, the Security Log (XSL) files and the System Error Log (ERR) files. In addition to these
4o files a header file should also be returned on each diskette or with each CBT test center's transmitted files.
a. Pro_ess CBT Transmission Files
[O416] CBT Transmission files may be received via modem to modem communications or by Banyan network trans-
45 fers from the test centers.
[O417] Figure 89 shows a preferred directory structure of the NDDS. Based on this directory structure, the transmis-
sion files to be processed by this option will be stored in the NDDS_RANS directoy when they are received from the
test center. These files are preferably in compressed format and are decompressed when they are moved to the NDDS/
WORK directory. Upon decompression the transmission file may contain at least one XSL file and a transmission
5o header file and may contain EPR and XRR files.
[O418] Transmission files are preferably read one at a time until all files in the NDDS_RANS directory have been
processed. A flowchart of this file processing procedure is shown in Figures 9OA through 9OC. The CRC of the com-
pressed transmission file is checked at 2O1 2. If the check fails an Exception report file message may be written at 2O1 4
and processing will continue with the next transmission file at 2O1 O. Table 1 6, below, lists possible error and warning
55 messages which may be written to the Exception Report file during various NDDS processes and the action preferably
taken when an error or warning condition occurs. 46
EP O 664 O41 B1
(PICTURE)
TABLE 1 6
5
1O
15
2O
25
3O
35
4O
45
5O
55 47
EP O 664 O41 B1
(PICTURE)
TABLE 1 6 (continued)
5
1O
15
2O
25
3O
35
4O
45
5O
55 48
EP O 664 O41 B1
(PICTURE)
TABLE 1 6 (continued)
5
1O
15
2O
25 [O419] If the CRC check is OK, the transmission file is decompressed at 2O1 6. The header file record counts are
then compared to the actual number of EPR, XSL and XRR files received at 2O1 8. If the counts for the EPR or XSL
files do not agree as determined at 2O2O, an Exception report file message may be written at 2O1 4 (Error Message
No. 9 or 1 O). If the count for the XRR files received does not agree a warning message is preferably written to the
3o Exception report file, but processing may continue with EPR and XSL file processing at 2O22 (Warning Message No. 1 ).
[O42O] Any DEMO EPR files are preferably moved to the NDDS/DEMO directory at 2O22 and an appropriate excep-
tion message is written to the Exception Report file (Warning Message No. 1 1 ). The XRR records are then preferably
moved to the NDDS/ERR directoy and the appropriate exception message is written at 2O26 to the Exception Report
File (Warning Message No. 1 2). The Process Count record is preferably written to the NDDS/HEADER directory at
35 2O28. Focusing now in Fig. 9Ob, the XSL files may then be processed at 2O3O creating skeleton EPRs for ''No Shows''.
The CBTN Database is then preferably updated at 2O32 with record version number changes contained in XSL records
so that the IPT, TPT, TPAK, TDA and administrative application version numbers being used at each test center may
be tracked for maintenance purposes. An exception message (Warning Message No. 7) is preferably printed upon
each update.
4o [O421] The EPR files are then processed by adding a record to the Examinee database or by adding the EPR file to
the Rejection Directoy at 2O38. (EPR processing will be described in detail below.) Then records may be added to the
Essay, Exception Report or Program Specific Data Files at 2O4O.
[O422] Afterthe last transmission file is processed as determined at 2O42 the NDDS/RESOLVE (Resolution directory)
is checked at 2O44 in Fig. 9Oc for any previously rejected EPR or XSL files that have been corrected. If there are any
45 corrected files, then file processing continues by returning to step 2O36. The XSL files are then preferably processed
by adding records to the Security/Event Log database and the Exception Report file at 2O46. Upon completion, the
program returns to the main menu shown in Figure 88.
b. Pro_ess CBT Data Disk(s)
5O [O423] The Data Disks are preferably received on 1 .44MB 3.5 inch or 1 .2MB 5.25 inch diskettes and there may be
multiple disks received from a test center for a testing day. The diskettes preferably contain at least one XSL file and
a header file and may contain EPR and XRR files. If multiple disks from the same test center for the same testing day
are received the header file is preferably stored on the last disk. A secondary screen, shown in Figure 91 , may be
55 displayed upon selection of the ''Process CBT Data Disk(s)'' option to prompt for initial and possible multiple test center
data disks. If F2 or any other key designated to initiate processing is pressed before a disk containing a transmission
headerfile forthe center being processed is loaded, the message ''HEADER FILE NOT FOUND. PLEASE LOAD NEXT
DISK FOR THIS CENTER'' is preferably displayed. 49
EP O 664 O41 B1
[O424] In a preferred embodiment, the data disks will also be processed one at a time. However, since multiple data
disks from the same test center for the same testing days may have been received, the processing shown in the
flowchart of Figures 92A and 92B are preferably implemented to process multiple disks. The files on the data disk are
Ioaded to the NDDS/WORK directoy at 2O5O. If there are additional data disks for that center, step 2O5O is repeated
5 until a data disk with a header file for that test center is found and the NDDS operator confirms that the last disk for
that center has been loaded at 2O52. The header file record counts are preferably compared with the actual number
of EPR, XSL and XRR files received at 2O54. If the counts for the EPR or XSL files do not agree an Exception report
file message is written at 2O58 and the next test center's disk(s) is loaded at 2O5O. If the count forthe XRR files received
does not agree a warning message is written at 2O62 to the Exception Report file and the program preferably continues
1O at 2O6O processing the EPR and XSL files.
[O425] The DEMO EPR files are then moved to the NDDS/DEMO directory and an appropriate exception message
is written to the Exception Report file at 2O6O. The XRR records may then be moved to the NDDS/ERR directory and
an appropriate exception message is preferably written at 2O64 to the Exception Report file. The Process Count record
is then written at 2O66 to the NDDS/HEADER directoy.
15 [O426] The XSL files may then be processed to create skeleton EPR records for ''No Shows'' at 2O68. The CBTN
Database is then updated at 2O7O with record version number changes contained in records in the XSLfile. Appropriate
Exception Report messages are written to the Exception Report File at 2O72. The EPR files are then processed and
records are added to the Examinee database or the EPR file is added to the Reject Directory at 2O74. Records may
then be added to the Essay, Exception Report, and/or data files at 2O76. The XSL files are then processed at 2O78 so
2O that records are appended to the Security/Event Log database and the Exception Repo_ file.
_. Pro_ess CBT Ba_kup Disk(s)
[O427] The Backup Disks are preferably received on 1 .44MB 3.5 inch or 1 .2MB 5.25 inch diskettes and there may
25 be multiple disks received from a test center. The diskettes preferably contain at least one XSL file and a header file
and may contain EPR and XRR files. In preferred embodiments, the Backup disk(s) may contain up to a month of
testing files. Therefore, if the ''Process CBT Backup disk(s)_.'' option is selected a secondary menu, shown in Figure
94, for file selection is preferably displayed. If the backup files for a test center are contained on multiple disks the
header file is preferably stored on the last disk.
3O [O428] To process CBT data files from a backup disk, the file(s) are preferably selected from the disk by date, date
range and/or examinee registration ID. In a preferred embodiment, individual files may be selectively processed by
using the Examinee Registration l D selection. The screen shown in Figure 93 may be displayed in a preferred embod-
iment so that files on multiple backup disks may be processed. In a preferred embodiment, the ''Process CBT Back
Up Disk'' option will be carried out according to the flowcharts shown in Figures 94A and 94B.
35 [O429] An NDDS operator will first be prompted to enter criteria upon which files to process can be selected at 2O8O.
The selection criteria may include, for instance the date, date/range and/or examinee l D registration number. The file
(s) are loaded at 2O82, based on the selection criteria, from the backup disk to the NDDS/WORK directory. If there are
additional backup disks for that center, step 2O82 is repeated until a backup disk with a header file for that center is
found and the NDDS operator confirms at 2O84 that the last disk for that center has been loaded. The DEMO EPR
4O files are then preferably moved to the NDDS/DEMO directoy and an appropriate exception messages is written to the
Exception Report file at 2O86. The XRR records are preferably moved to the NDDS/ERR directory and an appropriate
exception message is written to the Exception Report file at 2O88. The Process Count record may then preferably be
written to the NDDS/HEADER directory at 2O9O.
[O43O] Referring to Fig. 94B, the XSL files may then be processed creating skeleton EPR records for ''No Shows''
45 at 2O92. Appropriate Exception Repo_ messages may then be written to the Exception Report File at 2O96.
[O431] The EPR files may then be processed adding records to the Examinee database or the EPR file to the Reject
Directoy at 2O98. Other records may be added to the Essay, Exception Report, and data files at 2O99. The XSL files
may then be processed adding records to the Security/Event Log database and the Exception Report file at 21 OO.
5O d. Pro_egg Cou__ Fi_e _e_ord Pro_egg
[O432] As described above the process count file records are preferably written to the process count file during the
data or backup disk and transmission file processing. A count record for each test center's records that are processed
will be written as shown at 2O28 in Figure 9OA, at 2O66 in Figure 92A, and at 2O9O in Figure 94A. The counts in these
55 records that will be written during the data or backup disk and transmission processes will be the number of EPR's,
XSL's, DEMO EPR's and XRR's. Some examples of the process count file fields may include_.
TEST CENTER NUMBER 5O
EP O 664 O41 B1
EPR INPUT COUNT
XSL l NPUT COUNT
DEMO EPR l NPUT COUNT
XRR INPUT COUNT
5 NO SHOW EPR GEN COUNT
XSL PROCESSED COUNT
XSL REJECTED COUNT
EPR PROCESSED COUNT FOR EACH TEST PROGRAM
EPR REJECTED COUNT
1O e. End of Day Pro_ess
[O433] After all of the records are received from the test centers and processed by the NDDS, the End of Day Process
may be selected from the main menu shown in Figure 88. The End of Day Process generates an NDDS Processing
15 Control Report. The NDDS Processing Control Repo_ essentially provides a human readable form of the process
count file. The CBT test center operations personnel may review this report to verify whether all of the transmitted
records were received from the test centers.
2. Se_urity_Event Log Data Component
2O [O434] The security event log data processing component receives XSL records from the file processing component.
Specifically this process is initiated by the file processing procedures as shown at 2O46 in figure 9OC, at 2O78 in figure
92B, and at 21 OO in figure 94B. This process will preferably first verify the checksum of the XSL records. The files
resulting in error will be added to a Reject file and an appropriate message will be written to the Exception Report file.
25 The other records are added to the Security/Event Log database. The following are some examples of conditions which
might produce security exception messages in preferred embodiments.
1 . The occurrence of security violations.
2. A start rejection record was written as a result of the administrator being unable to properly log on.
3O 3. A system logon time occurred at an unauthorized time.
4. The testing workstation was open for greater than a predetermined maximum number of hours without being
closed.
5. The Start Log PC Histoy reflects an entry for which there is no corresponding XSL Start Log record.
35 These conditions will be checked for and an appropriate warning message will be written to the Exception Report file
for each record meeting the above criteria.
[O435] Each XSL file preferably generates one new master record, a number of event records, depending on their
event record types and a reconciliation record or records in the Security/Event Log Database. The master record is
written to the CBTN database and contains linking information to the other records in the database. The reconciliation
4O record preferably contains system generated and administrator entered test administration and demonstration session
counts and optional administrator text associated with each count group. These records may be used to generate
Exception Report file records for count discrepancies between the system generated and administrator entered counts
and for any administrator text returned in the record. Preferably, the printed Exception Report messages will subse-
quently show the counts and the text from these records.
45 [O436] Table 1 7, below provides a preferred structure of the Security/Event Log Database records which may be
created during the security/event log processing component.
(PICTURE)
TABLE 1 7
5O
55 51
EP O 664 O41 B1
(PICTURE)
TABLE 1 7 (continued)
5
1O
15
2O
25
3O
35
4O
45
5O
55 kThese event records have been described in Section IV(B) hereinabove.
52
EP O 664 O41 B1
(PICTURE)
TABLE 1 7 (continued)
5
1O
15
2O
25
3O
35
4O
45
5O 3. E_a_i_ee _e_fo__a__e _e_o_d (E__) __o_eggi_g
[O437] This process begins with duplicate record checking of the incoming EPR and Resolution EPR files. Any oc-
currence of an exact duplicate EPR file with an existing examinee database record will preferably post a warning
message to the Exception Report file and that EPR file will not be processed. This process will also check the EPR
55 files against the _BT_ database for test delivery software version number discrepancies. Additionally, record check-
sum, improper EPR sequencing and missing, invalid or undefined EPR conditions are also checked. Records with
these conditions will generate appropriate Exception Report file warning or error message records and will also cause
53
EP O 664 O41 B1
the EPR files with errors to be written to the Reject file directory. The EPR's that pass the above checks will preferably
generate new records in the Examinee database.
[O438] The E PR files will also be checked for EPR's containing essay response data. The essay EPR will be checked
for the information required to produce the appropriate essay form. If the essay E PR is missing required data or contains
5 invalid data an appropriate warning message will be written to the Exception Report file. This process will then extract
the fields required to generate a handwritten or a typed essay form and write an essay record to the Essay file. Essay
file processing is discussed in more detail below.
[O439] At the beginning of evey testing session and evey restarted testing session the test administrator can pref-
erably enter free form text and have that text recorded in the EPR file. This text may be written to the Exception report
1O file and reported in the Exception Report for that day.
[O44O] Demonstration EPR records are preferably copied to a Demo directory by the E PR process and will not un-
dergo any additional NDDS processing other than an Exception Report file record being written that notes the record
was received. Table 8 below lists some examples of Examinee Database records created during the Examinee Per-
formance Record File Processing.
15 (PICTURE)
TABLE 1 8
2O
25
3O
35
4O
45
5O
55 54
EP O 664 O41 B1
(PICTURE)
TABLE 1 8 (continued)
5
1O
15
2O 4. Format Post Pro_essing Component
25 [O441] The Format Post Processing Component allows testing programs to specify the format of the file they will
receive as input to their postprocessing database. This allows them to use existing databases of examinee results from
paper-and-pencil tests to store results from computer-based tests. Using a definition file as input to this process the
participating testing programs may select the order and EPR fields they wish to receive in their postprocess file or files.
3o This process may also do the appropriate data translation and conversion to ASCll of the EPR fields selected. The file
definition concept provides the ability to truly customize output files that can be changed at anytime with little or no the
need of NDDS program coding changes. The CBTN database master file is also an input to this process providing
center information that may be required for the postprocessing systems custom formatted record(s). In the case of
essay scores and reader information, this data may need to be merged in the program specific post processing systems
35 prOCeSS.
5. Reporting Component
a. A_tivity Report
4O [O442] Activity reports may be produced on a periodic basis, such as daily, weekly or monthly basis. These reports
may be produced from the data and calculations on data contained in both the Examinee and SecuritytEvent Log
databases with additional information provided by staff who manage the network of test centers for test time and test
fees. The daily reports may be produced bytest centers and showthe number and types of tests and income associated
45 With eaCh. They Can aISO inCIUde a CUmUlatiVe repO_ fOr different typeS Of CenterS, SUCh aS thOSe managed by fran-
chisees or educational institutions. Additional daily, weekly and monthly reports can generated based upon test center,
a date or a date range when processing occurred, or the date or date range when the test was administered.
[O443] When the ''Activity Reports'' option is selected from the main menu shown in Fig. 88, a secondary screen
prompting the ''NDDS'' operator for the report's criteria should be displayed. Fig. 95 provides one example of such a
5o screen. In a preferred embodiment, a daily activity report will use the current date as a default value. Fig. 96 provides
a flowchart of the activity reporting process according to this preferred embodiment.
[O444] Referring to Fig. 96, the activity reporting process first checks whether the NDDS operator has selected the
daily default reporting option. As shown in the example screen of Fig. 88, this can be done by simply pressing the
enter. If specific criteria has been keyed in by the NDDS operator the criteria is checked at 21 52 to determine whether
55 it is valid. For instance, if the cumulative center summary option is keyed, the NDDS operator keys in either a Y or an
N. If the date or dates are keyed, they are preferably entered in an appropriate format such as two digits for the month,
two digits for the day and two digits for the year and the year should not be greater than the current year. If the data
keyed in does not conform tothe edits displayed bythe screen shown in Fig. 96, a message for instance, ''l NVALl DATED
55
EP O 664 O41 B1
KEY. PLEASE CORRECT'' is preferably displayed. After such a message is displayed at 21 54, the process preferably
returns to 21 52 so that the NDDS operator may key in new criteria.
[O445] If either default reporting has been selected at 21 5O or criteria has been correctly keyed in at 21 52, the security
Iog and examinee performance record databases are checked at 21 56 to determine whether or not they both exist. If
5 either one is found not to exist at 21 56, an error message is preferably displayed at 21 58 and the process returns
control to the main menu. If both databases exist, then they are both checked to determine whether or not either is
empty at 21 6O. If either the security log database or the examinee performance record database is empty, an error
message is preferably displayed at 21 62 and control is returned to the main menu.
[O446] Where both the security log and examinee performance record databases exist and are not empty, the process
1O continues at 21 64 by calling a centeractivity report program and/or a cumulative center summay activity report program
depending upon the selections and entries made to the menu shown in Fig. 96. Preferably the default values and/or
the keyed in criteria will be passed to those programs at 21 64. These programs are standard database report programs
that gather the data specified by the query from the CBTN Database and format the report for printing in a human
readable form. Commercially available products, such as PROSORT OR BTRIEVE may be used.
15 [O447] In a preferred embodiment, both the center activity report program and the cumulative center summary activity
report program will return the following status codes.
O -- successful report
1 -- unsuccessful report
2O 2 -- no records in range to report
3 -- no data on file or software error
The return status is preferably displayed for the NDDS operator. If other errors occur, such as the commercial database
search program being used is not loaded, messages should preferably be displayed on the reporting menu screen.
25 b. Audit Trail Report
[O448] This report may be produced upon request and be generated from information in the Examinee database.
The purpose of this report is to be able to track any examinee information from the testing session to the time his or
3O her testing data, i.e. from EPR files, is turned over to the postprocessing system. When the ''Audit Trail Reports'' option
is selected from the main menu shown in Figure 88, a secondary screen prompting for selection criteria may be dis-
played. Figure 88 provides an example of such a screen.
[O449] There is no standard report produced for the Audit Trail report. However, at least one selection criteria field
should be keyed to produce a report. The following are examples of possible audit trail reporting selection alternatives.
35 1 . Examinees with a specific registration l D.
2. Examinees with a specific name.
3. Examinees taking a CBT at a specific test center.
4. Examinees taking a CBT at a specific test center on a specific test date.
4O 5. Examinees tested for all centers for a specific date range.
6. Examinees with a specific registration l D tested at a specific test center within specific date range.
[O45O] A flowchart implementing the Audit Trail Reporting process according to a preferred embodiment is shown in
Figure 98.
45 [O451] If specific criteria has been keyed in bythe NDDS operatorthe criteria is checked at 21 7O to determine whether
it is valid. For instance, If the data keyed in does not conform to the edits displayed by the screen shown in Fig. 95, a
messagefor instance, ''l NVALl DATED KEY. PLEASE CORRECT'' is preferably displayed at 21 72. Aftersuch a message
is displayed at 21 54, the process preferably returns to 21 7O so that the NDDS operator may key in new criteria.
[O452] If the selection criteria has been correctly keyed in at 21 7O, the security log and examinee performance record
5O databases are checked at 21 74 to determine whether or not they both exist. lf either one is found not to exist, an error
message is preferably displayed at 21 76 and the process returns control to the main menu. If both databases exist,
then they are both checked to determine whether or not either is empty at 21 78. If either the security log database or
the examinee performance record database is empty, an error message is preferably displayed at 21 8O and control is
returned to the main menu.
55 [O453] Where both the security log and examinee performance record databases exist and are not empty, the process
continues at 21 82 by calling an Audit-Trail-Report Program. Preferably the keyed in criteria is passed to the
Audit-Trail-Report Program at 21 82. The Audit-Trail-Report program is preferably a standard database report pro-
gram which gathers the dataspecified bythe selection criteriafrom the CBTN database andformatsthe datato generate
56
EP O 664 O41 B1
a human readable report. Commercially available products such as PROSORT or BTRl EVE may be used.
[O454] The following status codes are returned by the Audit-Trail-Report Program in a preferred embodiment_.
O -- Successful report
5 1 -- Unsuccessful report
2 -- No records in range to report
3 -- No data in file or a software error
Preferably the returned status and any other error are displayed for the NDDS operator before returning to the main
1O menu shown in Figure 88.
_. Daily Pro_essing Control Report
[O455] This report may be produced during the NDDS End of Day process and lists appropriate input, processed
15 and rejected record counts by test center. lt is prepared by counting the number of Examinee Performance Records
received ('Inputs') and classifying them as 'Processed' (successfully processed EPR files) or 'Rejected' (files requiring
resolution). This report is used to track the location and status of examinee performance files and resolve any discrep-
ancies in counts of records received from test centers and counts of records transferred to testing programs' post-
processing databases.
2O d. E__eption Report Generation
[O456] The Exception Report Messages can occur during any of the processing stages of the NDDS operation. When
an exception occurs, an Exception Report File is written. This reporting process produces the exception reports from
25 the messages accumulated in the Exception Repo_ File. lt should be understood that such reports, like the Activity
and Audit Trail Reports may be generated based on a variety of selection criteria, e.g., test center number, date,
message, etc.
[O457] In a preferred embodiment, the Exception Report Process would be implemented similarlyto eitherthe Activity
Report or Audit Trail Report Process except that Exception-Report-Program would be called instead of the programs
3O shown at 21 64 of Figure 96 or at 21 82 of Figure 98, respectively.
e. Se_urityIEvent Log Report
[O458] This report may be generated from information contained in both the XSL and examinee performance record
35 databases. In a preferred embodiment an XSL report is produced for all test centers. When the ''Security/Event Log
Report'' option is selected from the main menu, a secondary menu is preferably displayed. Figure 99 is an example of
a secondary menu that may be displayed prompting the NDDS operator for selection criteria.
[O459] In a preferred embodiment, no selection criteria is required. Rather a default report may be generated for the
current day. However, if specific reporting selection criteria is desired the following are examples of some possible
4O alternatives.
1 . Report for a single test center for the date the XSL and EPR files were processed.
2. Report all test centers for a previous date.
3. Report all test centers for a previous date range.
45 4. Report a single test center for a previous date range.
[O46O] It should be understood that this process is preferably implemented according to the flowchart shown in Figure
97 for the Activity Report, except that the Security/Event Log Report Programs would be called at 21 64 in place of the
Activity Report Programs.
5O 6. Essay File Pro_essing
[O461] During the CBT file process the NDDS preferably writes an essay record into the NDDS Essay File for evey
EPR that contains Examinee Response records. The EPR file for these examinees preferably contains two Essay
55 response records. These correspond to the two topics presented to the examinee during the essay test. The EPR
should be written to the Reject directory if one or both records are missing. One of the records preferably contains the
essay text that is to be scored with its topic number while the other response record contains the topic number of the
alternate topic text presented to the examinee and possibly essay text that is not to be scored.
57
EP O 664 O41 B1
[O462] When the Essay Printing option is selected from the Main Menu the NDDS produces, based on the Essay
Type indicator in NDDS Essay file, either a Handwritten or Typed essay form for each record in the file. To print the
text for the essay topics this process searches the NDDS Topic file by Topic Number for the text to print for the Essay
Topic and Alternate Essay Topic fields on the forms.
5 [O463] These forms may be used by the essay readers and returned to the appropriate postprocessing system for
further processing. Should there be a need to reprint these forms the Examinee Essay files will be archived and saved
by the NDDS.
7. Reje_UResolution pro_essing _omponent
1O [O464] During both the SecuritytEvent Log and File Process components, records can be written to the Reject file
directory. The files in this directory are copies of the original EPR or XSL files received from the test centers. EPR and
XSL files that are written to the Reject file are generally not processed successfully by the system and, therefore, the
data from those rejected files is not written to the Examinee or SecuritytEvent Log databases. However, those records
15 are counted in that day's Daily Processing Control Report as rejected EPR or XSL files. ln a preferred embodiment,
the files in error that are written to the Reject file will require an individual case by case manual resolution process.
Records may however be corrected by other means (editors, etc) and moved to a resolution directory for processing
during the next NDDS processing run.
[O465] As mentioned above files that actually match existing examinee database records (e.g., the same Registration
2O number, Full Name and Time/Date stamp) are preferably not processed. It is further preferred that these records will
also not be written to the Reject file since it should be assumed that these records were mistaken retransmissions of
records already received and processed. Additionally, EPR files received that have the same registration number and
full name as an existing examinee record but have a different time/date stamp, or EPR files with the same registration
number but a different full name as an existing database record are preferably processed as separate examinee
25 records.
8. CBTN Information Component
[O466] This component consists of an application for adding, updating and deleting the CBTN database records. The
3O NDDS may access this database to check the test delivery system version number information and also update the
database with version numbers supplied by the XSL file Maintenance Performed records. The NDDS will preferably
use the CBTN database as a list of current test centers with their transmission schedules and will check daily trans-
mission and diskette receipts against that list. In preferred embodiments, warning message records are written to the
Exception Report file for the test centers from which no records were received, but were scheduled or for files received
35 at a time transmission was not scheduled.
[O467] In a preferred embodiment, the operation of the CBTN database may be described as follows. It should be
understood that functional key assignments and valid entry criteria described below are purelyfor explanatory purposes
and that the invention is not to be limited thereto. The CBTN application is entered from the NDDS main menu screen.
The first screen displayed is the signon screen shown in Figure 1 OOA. The function of this screen is to allow entry of
4O the user's ID which is preferably verified against a valid user ID table. After acceptance of a valid user ID, a second
screen may be presented as shown in Figure 1 OOB. This screen may be used to select or add a test center's record
to the CBTN database. To add a new CBTN database record, place the cursor on ADD CENTER and depress the key.
Preferably a blank formatted screen such as that in Figure 1 OOC will then be displayed. To display the information for
any other record on the screen listed in Figure 1 O1 C, the cursor should be placed on the desired record and the enter
45 key depressed.
[O468] The X's in the screen fields shown in Figure 1 OOC indicate that any character is allowed. The 9's indicate that
only numeric data should be entered. The AUTH field is a O or 1 and the Transmission Schedule field is a Y (yes), N
(No) or D (Diskette).
[O469] The CBTN Center, Address, City, State, Zip, Version Information and Transmission Schedule fields are pref-
5O erably entered while, all other fields may be optional. F2 is used to Save the record, F3 is used to Delete the record
and ESC is used to Cancel Changes on CBTN processing screens where applicable.
[O47O] Depressing the F4 key from the screen shown in Figure 1 OOC preferably invokes an administrator selection
screen shown in 1 OOD, which provides information about the administrators at that test center and is used for example
to interpret the logon l Ds recorded in the security log or to track which administrators are at which test centers. De-
55 pressing the F5 key from the test center screen preferably brings up a Package Control lD selection screen shown in
Figure 1 OOE which provides information aboutthevarioustesting program packages and theirversions that are installed
at the center. When the F6 key is depressed from the test center screen a comment selection screen shown in Figure
1 OOF is preferably displayed which allows staff to review the comments entered by administrators during testing ses-
58
EP O 664 O41 B1
SlOnS.
[O471] The ENTERED BY field shown in Figure 1 OOF is preferably inserted by the application from the User l D keyed
in on the Logon screen. The current date is also displayed in the DATE field. The PROGRAM CODE field prompts for
the program code designating a specific test, e.g. SAT, GRE, etc. The AIR (indicates that Administrator has provided
5 a paper report describing some occurrence during a test session) and ANNOT (indicating that the administrator had
made an annotation during the Close-of-Day Process) preferably requires a Y or N. The EVENT CODE and CAT
(category) fields may be checked against atable of allowable choices for these fields, which can be seen by depressing
the F1 key. Allowable characters for the ACTlON IND field are preferably R and T and the ACTlON BY field is the
Logon l D. The COMMENTS and NOTES are free form non edited enty fields.
1O [O472] Acomputer based testing system and a method of computer based testing have been described. The following
appendices have been provided to further supplement the detailed description by providing exemplary pseudo code
and flowcharts for several of the procedures described herein implemented by the computer based testing system of
the present invention.
15 Appendix A -
Pseudo Code and Corresponding Flowcharts for the Item Preparation System
Appendix B -
Pseudo Code and Corresponding Flowcharts for the Item Preparation Tool
Appendix C -
2O Pseudo Code and Corresponding Flowcharts for the Test Packaging Application
Appendix D -
Pseudo Code and Corresponding Flowcharts for the Test Delivery Application
[O473] While the invention has been described and illustrated with reference to specific embodiments, those skilled
25 in the art will recognize that modifications and variations may be made without departing from the principles of the
invention as described hereinabove and set forth in the following claims.
3o ______ _
ea___o Cod_ for kb_
_t_ _r___r_t_o_ _oo_
35
4O
45
5O
55 59
EPO664O41B1
A-_
___-_roGad_r__
5 A _lowc_art depicting the _IN-PROCED_ is shown in Pigure
A-l.
loo Display the main menu a_ shown in. _igure O.
1o 2OO InitialiZe the _i_eCtiOns _UbsyBtem.
3oo Petch the next event _rom the event queue, I_ the
event is _-PROG_, continue at 5OO; else,
dispatch to the event's processing routine and
15 continue at step 4OO.
qoo I_ there is an item on screen per_orm the
Check Integrity Procedure (described below)
2O h- . - _ ,t t h h d
5OO I_ t e item Or any O i s parame erS aVe C ange ,
display a query asking the user i_ s/he wants to
save the changes. I_ so, save all in_ormation,
25 6oo Exit to the operating gy8tem,
3O
35
4O
45
5O
55 6O
EPO664O41B1
Icon Events A-2 . Icon Events
ICO_-_. _
5 A flowchart of the ICON-NEXT procedure i8 ehown in Figure A2.
_oo Perform the Check-Integrity-Procedure (described
. below).2OO
,, If the item or any of its parameterG have changed,
display a que_ asking the user if s/he wants to
save the changes. If Go, save all info_ation.
3oo Load the dieplay panes with the contents of the
15 next item (the contentG of the panes and pane
arrangements are assigned by the PRESENTATION
procedures described below)
2O 3_o E bl th p _ on
na e e reV iC .
Qoo If this is the last item in the batch, disable the
Next icon.
25 _ooo Return to Caller.
ICO_-__. _
A flowchart of the ICON-PREV procedure is shown in Figure A3.
,, _oo Perform the Check-Integrity-Procedure (deGcribed
below).
2oo If the item or any of its parameters have changed,
display a query asking the user if s/he wants to
35 save the changeG. If so, save all information.
3oo _oad the diGplay panes with the contents of the
previous item Cthe contents of the panes and pane
4O arrangementg are assi_ed by the pRESE_ATIoN
procedures deGcribed below)
qoo If this is the firet item in the batch, disable the
45 Prev icon.
_ooo Return to caller.
ICO_-___:
5o ICO_-__.
ICO_-R_.
_oo Return to caller.
55 61
EPO664O41B1
Icon EventS A-3 Icon Even__
ICO_-__
5 A _lowcha_t o_ the I_O___K procedUre ig 8ho_ in _i_re
A-Q(A3.
._OO I_ the 'marked' image o_ the Mark icon is current
1o di8played, di8play the 'Unmarked' image, else,
' display the 'ma_ked' image.
_OO Return to caller. '
,5 ICO_-Q_IT _d ICO_-_IT_
A _lowchart o_ these procedures is shown in Pigure A-4(B).
_OO Post event _-PROG_
2OO Return to caller.
2O
25
3O
35
4O
45
5O
55 62
EPO664O41B1
Internal Menu Events A-_ Internal Menu Events
IP-G_-8_: _
5 A _lowchart o_ this procedure is ehown in Pigure A-5(A).
_oO Disable each main menu option speci_ied by the
event parameter.
,o 2oo Return to caller.
IP-_G__-__
A _lowchart o_ thîs proced_re is _hown in Pigure A-5(B).
_oo Enable each main menu option speci_ied by the event
15 parameter.
2oo Return to caller.
x-___AG8_
2O A _l h t _ this p edur ' gh ' p' re A 6
owc ar o roC e iS oWn in lgu - .
_oo Zliminate the main menu.
2oo Petch the _ile name to display _rom the list using
25 the index passed as a parameter to the event.
3oo Display the speci_ied message.
4OO I_ the message was _n t terminated via the special
3o key eequence.
- p_ost
'' event M-MESSAGE -- parameter iG index +
-' continue at 6OO.
35 5OO Post event M-_IN
55O Post event M-PI_E-OPEN
6oo Return to caller.
4O x-_I___
A _lowchart o_ this procedure is shown in Pigure A-7.
_oo Display the main menu _
45 2OO Return to caller.
5O
55 63
EPO664O41B1
Internal Menu _vent_ A-_ Inte_nal Menu Even__
__-_O___T_. _
5 A _lowchart o_ thiG procedure is shown ie Pi_re A-8.
_OO __ state is C_TING-ECE_
- I_ the user has not entered the numbe_ o_ keys
1o de_ined Via the 'NU_er O_ Reqlired ReSpon_eG'
ent_ in _i_re _2 during event M-RE_PoN_E
- display an error message
- continue at _OOO.
15 - Post event M-KE__-EDIT.
- Continue at _OOO.
2OO I_ state is CREATING _GE LO_R or
2O - -
CREATING-_GE-UPPER;
- post event M-___-EDIT
- continue at _OOO.
25 _oOo Return to _aller. '
3O
35
4O
45
5O
55 64
EPO664O41B1
Menu Option Events A-6 Pile Option
_-___-A8O__ _
, A _lowchart o_ this procedupe is shown in Pigure A-9.
1OO Display the dialog bos graphically depicted in
_igure A-9(A) (the 5ign Off button ie non-
functional).
1O _oo _a.t for the uger to click on oK
i .
3oo Retu_ to caller.
_-____-O__.
,, A flowchart of this procedure is shown in Pigures A-tO
through A-_4.
5O Set search directory to current working directory
1OO Display the screen graphically depicted in figure
2O A_1o(A).
2OO _f this event was per_ormed previously, set file
extension to the last one used; else, set file
2, eXtenSiOn tO _STE.
3OO Search apecified directory for filea matching
currently 8elected extension and diaplay in the
Files section of the dialog.
3O .
4OO Respond to user input as follows.
- Open Button.
' - If no file is selected, display error
3, meSsage and COntinUe at 4OO.
- Process the Open request according to the
file type currently selected as follows.
- __TE
4O - set state to ITEM_
- disable the 'Batch Item Components'
aelection of the 'View' option in
4, the main _enU
- read in data for the selected item
and load the item panes with display
data (the item will be diaplayed
5O h th. d. l d )
w en is ia og en s
- continue at 1OOO.
- .I_
,, - 8et state to B_TCHES
65
EPO664O41B1
Menu Option Events _-7 Pile Option
- show the icons
, - enable the 'Batch Item ComponentG,
' eelection o_ the 'View' option in
the mai_ menu
- load all _ile names in the batch
1O d . d _ h _.
- _ea tn ata Or t e iret ttem kn
the batch and1oad the item eane_
with display data (the item will be
1, displayed when this dialog ends)
- disable the Prev icon
i_ the batch contains more than one
item, enable the Mext icon; else,
2O disable it
- continue at lOOO.
- .DIR
2, - set gtate to DIÆ_TIO__
- dieable the 'Batch Item Components'
selection o_ the 'View' option in
the main menu
3O - Per_o_ event OS_-DIRECT --
subevent DIRECT-REQUE_T -_ in the
Directions Subsyatem, now, speci_y
35 the gelected di_ectiong _ile _or
display
' - Continue at lOOO.
- GI_ and M5G
4O ' '
- get state to ME_5AGES
- i_ .GI_, 8et mesgage number to 25;
elae, isolate message numbe_ _rom
45 ' the lagt two charactera o_ the _ile
name
- save the _ile name o_ the .GI_ op
,o .M_G in a li_t at the index number
determined in the step above
- post event M-ME5SAGE -- gubevent
M-ME_SAGE -- parameter is the
55 meggage nu_er to display
66
EPO664O41B1
Menu Option Events A-8 Pile Option
- continue at _OOO.
5 - .TOP
- hide the icons, i_ displayed
- set state to E__A_
- disable the 'Batch Item Components'
1O l . _ ,
se ectiOn o the 'View' option in
the main menu
- display the text oY the Gelected
15 topic, wait _op use_ to dismiss
display
- _ continue at 4OO.
- Cancel Button.
2O - continue at _OOO.
- Items Radio Button.
- _et _ile exten8ion to ._T_
25 - Continue a_ 3OO.
- Batchee Radio Button.
- _et _ile extension to .I_
- Continue at 3OO.
3O - Directions Radio Button.
- Set _ile extension to .DIR
- Continue at 3OO.
35 _ GI_ _adio Button..
- Set _ile extension to .GIS
- Continue at 3OO.
4o - Message8 Radio Button.
- Set Yile extension to .M_G
- Continue at 3OO.
- Essay Topics RadiD Button.
45 * Set Yile extension to ._op
- Continue at 3OO.
- Directo_y
5o - Change current search directo_ to the
speciYied directory.
- Continue at 3OO.
_ooo Remove the dialog di8played in _t.ep _OO
55 __oo Return to _aller. 67
EPO664O41B1
Menu Option Events A-_9 Pile Option
M-__k_-GAV8_ .
, n _lowchart o_ this procedure is shown in Pigure A-_5(A).
_Oo I_ an item ie currently displayed
- i_ the item or any o_ ite parameters have
,o _hanged, dieplay a _e_ aSking the USer i_
s/he wants to save the changes; i_ so, eave
all in_ormation
,, 2OO Return to caller.
M-_I__-_R_:
A _lowchart o_ this procedure is _hown in Pigure A-_5(B).
_OO Print the 8cree_ to the currently selected printer
2O 2OO Return to caller.
25
3O
35
4O
45
5O
55 68
EPO664O41B1
Menu Option EventS A-_O View Option
X-VI_-____ .
5 A flowchart of thie proceudre ie shown in Figu_e A-l6.
_oo Display the dialog graphically depicted in figure
4.
_oo Accept user input and process as followG.
1O _ oK
- continue at 1OOO
- Print
,5 - print a summary of the information
contained in the dialog on the printer aG
per figure 3(a)
- continue at 2OO.
2O _ooo _ th d, l g d_gpl d _ _te loo
e_OVe e ia O k aye in p
l_oo Return to caller.
M-COM_O_-__ST_
25 A f1owchart of this procedure is shown in Figures A-17 to
A-l8.
1oo Scan the current working directo_ _or .CT_ files.
_oo Por each .C_k file, extract the .ST_, .R_P, .D_R,
3O and ._Ep _i.le re_erenceg contai_ned therei.n.
3oo Dieplay the identification numbers of each file
t_e referenced in each .CT_ file in the scrollable
35 area of the dialog graphically depicted in figure
A-l7_A_.
_oo Accept u_er input and process as follows.
- OK
4O _ conti.nue at 1ooo,
- Print
- print a eumma_ of the information
45 contained in the dialog on the printer as
per figure A-_7(8)
- continue at 4OO.
- Stem
5O _ gort the lineg digplayed i_' the
ecrollable area by identification number
of the ._TZ file '
55 - redisplay the scrollable area in sorted
69
EPO664O41B1
Menu Option Events _-__ View Option
order .
- continue 4OO
5 - Stimulus
- sort the lines displayed in the
scrollable area by identi_ieation number
1O o_ the ._p _i'le
- redisplay the 8crollable area in sorted
order
15 - continue 4OO
- Re8ponse
- sort the lines dieplayed in the
scrollable area by identi_ication number
2O o_ t_e .R_p _i.le
- redi8play the 8crollable area in sorted
order
25 - continue 4OO
- Direction8
- sort the linea displayed in the
8crollable area b_ identi_ication number
3O o_ tbe .D_R _i_le
- redisplay the scrollable area in sorted
order
35 - continue 4OO
1ooo Remove the dialog displayed in atep 3OO.
11oo Return to caller.
_-gK__-_O_O_-_I_T;
4O A _lowchart o_ thie procedure ia ahown in Pi_res A-19
through A-21.
2oo Por each .CT_ _ile e_umerated in the batch list,
45 extract the ._TE, .RSP, .D_R, and .ÆP _iles
re_erences and keys contained therein.
3oo Diaplay the order withfn the batch, identi_ication
number8 o_ each _ile t_e re_erenced in each .CT_
5O _ile, and anawer keys in the acrolla'ble area o_ the
dialog graphically depicted in _igure A-19(A).
Qoo Accept user input and proceas as _ollows.
55 - OK 7O
EPO664O41B1
Menu Option Event_ _-_2 View Option
- continue at _OOO. .
5 - Print .
- print a summary o_ the in_o_ation
contained in the dialog on the
printer as per _igure _-2O(A)
1O _ continue at qoo.
- Print All _
- print a summa_ o_ the information
15 contained in each .CT_ file
referenced by the batch list on the
printer as per figures A-2O(B) and
A-2o(C)
2O .
- continue at 4OO.
- Batch Order
- sort the lines displayed in the
25 scrollable area by batch order number
- redisplay the scrollable area in sorted
order '
- continue at 4OO.
3O - Stem
- sort _he lines displayed in the
scrollable area by identification number
35 of the .STE file
- redisplay the scrollable area in sorted
_ order
- continue 4OO
4O - Stimulus
- sort the lines displayed in the
scrollab1e area by identification number
45 of_ the .REP file
- redi8play the scrollable area in sorted
order
- continue QOO
5O - Reeponse
- sort the lines displayed in the
ecrollable area by identification
55 _umber of the .RSP file
71
EPO664O41B1
Menu Option Events A-_3 View Option
- redisplay the scrollable area in
5 8orted order
- continue 4OO
- Directions
,, ' - sort the 1ines displayed in the
scrollable area by identi_ication
number o_ the .DIR _ile
- redisplay the 8crollable area in
15 aorted order
- continue 4OO
_ooo Remove the dialog diGplayed in step 3OO.
2O __oo _ t t ll
e urn O Ca er. _
25
3O
35
4O
45
5O
55 72
EPO664O41B1
Menu Option _vent8 _-_4 Components Option
_-_O_O_-_____O_S.
, _ flowchart of this procedure is shown in _iguies A-22
through A-24.
1OO _iaplay the dialog graphically depicted in _i_re
A-22(__). Pill the list with all .DIR _iles _ound
1O i_ tbe cu_pent woiking directo_. _igable the
Popup Window radio button.
2oo If _irections aie already selected for this item,
,, highlight the .DIR _ile u8ed in the list and 8et
the radio button coiresponding to the type
previously selected.
3oo Accept uGei input and process as follows.
2O _ oK
- I_ the None radio button is selected.
- clear the Direction8 component from the
2, current item
- continue at 1DOO.
- I_ the user did not select a .DIR file
_rom the list. _
3O - di.gplay an error meggage
- continue at 3OO.
- If the Pixed Window radio button is
3, selected and the Number of _ines entered
by the u_er is zero.
- di8play an error message
- continue at 3OO.
4O _ _ave the gelected .DIR file name and
Display _e for the current item.
- Continue at 1OOO.
4, - Cancel
- Restore the parametei8 of the
current item to the values that
existed prior to step 1OO.
5O - Continue at 1OOO.
- None Radio Button
- Continue at 3OO
- Pixed Window Radio Button
55 73
EPO664O41B1
Menu Option Events A-__ Components Option
- Continue at 3OO .
, - _mbedded in Component Radio Button
- Continue at 3OO.
_ooo Remove the dialog di_played in _tep tOO
11oo Redisplay the current item with the directions t_e
1O and _ile 8peci_ied in the 8teps above using the
pre8entation _ormat previously selected using the
Pre_entation - Template menu option (described
15 below).
1_oo Return to caller.
M-CO_O_-_T___:
,, A _lowchart o_ thi8 procedure is shown in Pigure A-25.
1oo _isplay the dialog graphically depicted in _igure
25(A). Pill the lî_t with all .REF _iles _ound in
the current working directo_.
25 _oo __ _timulug ig already gelected _or thig item_
highlight the .REF _ile used.
3oo Accept u8er input and proces8 as _ollows.
3o - OK
- I_ the _timulus (.REF) _ile selected
is - _one - or the user did not
aelect a _timulu8 _ile
35 _ clear the_ _ti.mulus component _rom
the current item
- continue at tOOO.
4o - _aVe the SeleCted .REF _ile name _Or
the current item.
- Continue at 1OOO.
- Cancel
45 - Restore the parameters o_ the
current item to the valuee that
exi_ted prior to step 1OO.
,o - Continue at 1OOO.
tOOO Remove the dialog displayed in step tOO
ttoo Redisplay the current item with the _timulus
speci_ied in the steps above using the presentation
55 _o_at previously selected using the Presentation -
74
EPO664O41B1
Menu Option Events A-_6 Components Option
Template menu option (described below).
5 12oo Ret_rn to caller.
M-_O_O_-_5_O_5__
A flowchart of this procedure is shown in Pigure A-26.
1o 1oo Display the dia2og graphically depicted in figure
A-26(A). Pill the list with all .R5P files f_und
in the current working directo_.
,, 2oo _f a Responee is already selected for this item,
highlight the .RSP file used.
3oo Accept user input and process aG follows.
- OK
2O - _f the Response (.RSP) file selected is -
_one - or the user did not select a
Stimulus file
25 l th St_ l t f
- c ear e imU US COnlponen rom
the current item
- continue at 1OOO.
3o - 5ave the selected .RSP file name for the
current item.
- Continue at 1OOO.
3, - Cancel
- Restore the parametere of the current
item to the values that existed prior to
step lOO.
4O - Continue at 1OOO.
looO Remove the dialog displayed in Step lOO
llOO RediSplay the current item with the ReSponGe
45 specified in the eteps above using the preeentation
format previouelp selected using the Presentation -
Template menu option (described below).
,o 12OO Return to caller.
55 75
EPO664O41B1
Menu Option Events A-_7 Presentation Option
_ __g_-T__T_. _
5 A_elow_hart of _hiG procdure is gho_ in pigwres A___ th_ough
A-3O.
1oo _i_p_ay the dialog graphically depicted in _igure
1o A-27CA).
2oo _e the us.er previoualy aelected different optiona
using this dfal_g.
- highlight a template Ghown across the top
15 f th d. l box _ p.g _ 2_(A) get
O e la Og in i Ure - ,
the Pane _izing. and set the pane
organization previously selected.
2o ElSe
- highlight template #_, set pane sizing to
5O_, and set pane organization to display
all currently specified components in
25 pane #_ i_n the followi_ng deeault order..
_ Directions
- Stimulue
3o - Stem
- Respon8e
3oo Accept user input and process as follows.
- OK
35 - _f the _op _ ig e_al to O or the
Bottom _ is equal to O, display
error mesaage and continue at 3OO.
- If any pane does not have at least
4O one component a88igned to it,
display error meaaage and contin_e
at 3OO.
45 - _ave percentages. template and pane
organization with the item.
- Continue at 1OOO.
- Cancel
5O - Restore the parametere oe the
current item to the valuee that
existed prior to step _OO.
55 - Continue at 1OOO.
76
EPO664O41B1
Menu Option Events A-_8 Presentation Option
- Template Selection '
5 - Unhighlight the previously 8e1ected
template.
- _ighlight the newly selected
1o template.
- Re_et.,the pane organization as
followg.
- I_ a one pane temelate was selected,
15 list all currently eelected
components in the default order
graphical1y depicted in figure A-
2o 3O A).
- Tf a two pane template was selected,
list all currently eelected
components in the default order
25 graphiCally depicted in figure A-
3o(B).
- Tf a three pane temp1ate was
3o _eleCted_ liSt al1 CUrrently
8elected components in the default
order graphically depicted in figure
A-3oCC).
35 _ote.. the Di_reCti_ong component i_g
preferably displayed only if
Components - DireCtions wae selected
4o from the main menu and the 'Embedded
in Component' display t_e was
selected C8ee
M-COMPONENT-DTRECT_ONS).
45 _ Continue at 3oo.
- Move Component
- Move the highlighted component from
5o the current pane to the _pecified
pane. Figure A-3O(I7) andA-3O(E)
depict the before and after
displayed affected by moving the
55 Stimulu8 component from pane _ to
77
EPO664O41B1
Menu OptiOn EVentG A-_9 R_e_entation Option
pane 3. _
5 _ _onti.nue at 3oo.
1oOO Remo_e the dialog displayed in step 1oO
__OO Redisplay the c_rrent item with the Respon_e
1o Gpe_i_ied in the Btep_ above __ing the presentation
_o_at previouely 8elected u6ing the Presentation -
Template menu option.
__OO Return to caller.
15 _
_-____-_____.
_ flowchaft of this proCedure is ehown in Pigure A-31.
1OO Display the dialog graphically depicted in figure
2o A-31(_).
2oO Accept user input o_ Charactere defining
paraphrase.
3OO If dialog is canceled, restore previous. possibly
25 null, parap_a_e and continue at 1OOO.
4OO Store paraphfase with item.
1OOO Retu_ to caller.
3o _-__8_-_O______
A flowchart of this procedure is shown in Pigure A-32.
1OO Di8play the dialog graphiGall_ depfcted in figure
A_32(A).
35 f . . . l . .
2OO _ pO_itiOning Va UeS are 8peCified for the Curfent
item, display the values in the dialog; else,
display the d_fault valueG shown.
4o 3OO ACCept u8er inp_t _d gfOCe88 aS fOllOWa.
- OK
- Save the highlight and centering ID
with the current item
45 _ . Do
- OntinUe at 1 O.
- Cancel
- Reatore the parameters of the
5o CUrrent item tO the ValueS that
exi8ted prior to step lOo.
- Continue at 1OOO.
1OOO Remove the dialog displayed in step 1OO
55 1_oo Redi.gplay the current i.tem wi.th the _ti._uluG bloc_
78
EPO664O41B1
Men_ Option Events A-2O Pre_entation Option
, speci_ied in the steps above highlig'hted and
centered in the 5timulus pane. (Ignore i_ no
stimulus.)
__oo Return to caller.
1O
15
2O
25
3O
35
4O
45
5O
55 79
EPO664O41B1
Menu Option Events A-22 Re_ponse Option
M _a_O____ _
5 A__l h t _ th_ d re i_8 _ho _ F_ r A 33
OWc ar O tS prOCe U _ in i_ ee -
through A-36.
_oo _isplay the dialog graphically depicted in _i_re
1o A-33(A) -- setting the radio buttons and 'Number o_
Reaponses' to the va_lues previoualy de_ined _or the
itern.
2oo Accept u8er input and proce8s as _ollows.
15 - OK
- Save the _ollowing attributes with
the current item.
2o - Number o_ Required Responses
- Response Class
- Response _e
- Responae parametere declared during
25 display o_ the Params... dial Dg.
- Continue at _OOO.
- Cancel
3o - Restore the parameterS o_ the
current item to the values that
existed prior to step _OO.
_ - Continue at _OOO.
35 - pa_s...
- Di8play a dialog through which the
u8er enters parameter8 attuned to
4o the Re8ponse Class and Response Type
selected. CRepresentatfve dialog
_or single selection, multiple
choice graphicallv depicted in
45 _i_gure A_3__A).
- Hold parameters
- Continue at 2OO.
5o - Single _election Radio _utton
- _nable the _ollowing Reeponse T_e
radio buttone. _
- Multiple Choice
55 - _cale
8O
EPO664O41B1
Menu Option Events A-22 Response Option
, - Bar Chart/Histogram '
- Grid
- Zones
_ - Insert Text
1O - Disable all other_
- Continue at 2OO.
- Multiple _election Radio Button
15 bl ll .
- Ena e the _o owing ReGponse T_e
radio buttons.
- Mu1tiple Choice
2O - __ale
- Bar Chart/Histogram
- Grid
2, - ZOnes
- Ordering/Matching
- Disable all other8
- C_ntinue at 2OO.
3O - Pree Response Radio Button
- Enable the _ollowing Response _e
radio buttons.
35 _
- _u_eric Ent_
- Praction
- Di8able all other8
4o _ - Continue a_ 2OO.
- Re8ponse _e Radio Buttons
- _et the response t_e o_ the current
4, item to match th.at o_ the radio
button selected.
-_ Continue at 2OO.
_OOO Remove the dialog displayed in step _OO
5O __OO Redi8play the current item u8ing the response class
and type, number o_ required responses, and
parameter8 de_ined in the stepG above.
55 _2oo Return to caller. 81
EPO664O41B1
Menu Option Events A-23 Options Option
5 _-o___O__-__G____ '
A _lowchart o_ this procedu_e is shown in Pig_re A-37.
_oo __ integ_it_ che_king is in e__ect, turn it o__;
_ elge, turn it on.
1O _oo Return to caller.
15
2O
25
3O
35
4O
45
5O
55 82
EPO664O41B1
Menu Option _vents _-2_ Test Option
M _GT __GRI_; _
5 - - .
A _lowchart o_ thiS procedure is Shown in Pigure A-38(A3.
1oo Per_orm the Check-Integrity-Procedure.
2oo Retu_ to caller.
1o _-T_5T-__O_TI_G. '
A _lowchart o_ this procedure is 8hom in Pigure A-38(B).
1oo Check whether the u8er has chosen the number o_
respon_es apeci_ied _or thiS item via event
15 M-RE_PO__E. I_ not, display an in_ormational
screen noti_ying the user that all required
responseS have not been entered.
2o 2OO Return to caller.
M-TEST-SCOR__G.
A _lo_chart o_ this pro_edure is shom in Pigure A-39.
_OO Compare the user's testing answer with the
25 currently de_ined answer keys. Possible results
and actione _or each conditîon are as _ollows.
- SCORE UNAN__RED. (The number o_ required
3o respon_es as de_ined via the M-RESPON__
event i8 O, or no respon8e is selected.3
- Display an in_ormational me8aage
indicating that no responae has been
35 selected.
- Continue at _OOO.
- SCORE IMPROPER. (The nu_er o_ required
4o re8pOnses dOes nOt matCh the nUmber o_
responaes 8elected.)
- Display an in_o_tional mes6age
indicating that the incorrect number
45 o_ reSponses has been selected.
- Continue at 1OOO.
- ECE_ MISSING. (No scoring keys have been
5o entered Via the M_KE_S_* eVentS.)
- Diaplay an fn_o_ational mesSage
' indicating that no keys have been
de_ined _or the current item.
55 _ Conti_nue at _ooo.
83
EPO664O41B1
Menu Option Events _-2_ Test Option
5 _ _O_G. (The COrreCt nU_er O_ reSpOnBeS
has been supplied and key8 are de_ined,
but the selected re_pon8e is incorrect.)
,, - Di8play an info_ational message
indicating that incorrect responses
have been Sele_ted.
- Continue at _OOO.
15 _
- RIGHT. (The correct nUmber o_ responses has
been supplied, keys are de_ined, and the
selected responge is correct.)
2O - Digplay _ in_o_ational _eggage
indicating that correct responses have
been selected.
25 - COntinUe at _OOO_
_ooo Return to caller.
3O
35
4O
45
5O
55 84
EPO664O41B1
_-26
_-__-____; _
, A flowchart of thi8 procedure is shown in Pigures A-qo
through A-q3.
1OO CheCk Whether a reGponse t_e has been declared for
the current item via the M RESPONSE event. If not,
1O d. l -
iSp ay an errOr _essage and ContinUe at 2OOO.
2OO CheCk Whether the 8Core type for this item has been
set to 'Mumber o_ _elections for Key' via the
1, M-SCORE e_ent u8ing figure _-QO(A). If not,
continue at 4OO.
3OO _isplay the dialog graphically depicted in figure
_-4O(A) with 'Use Keys' deselected and 'number of
2O selections for key' set to O, Accept user responee
and proceee as follows.
- O_
25 - CheCk whether the uger haG gelected
'Use KeyS' or has entered a number
of selections for the key.
- If neither
3O - dieplay error me8Gage
- continue at 3OO.
- Else
35 - gave Gcore type Gelected in the
dialog With the current item
- conttnue at 1OOO.
- Cancel
4O - Restore the score t_e of the
CUrrent ite_ to the valueg that
existed prior to atep _oo.
45 _ Continue at _ooo.
4OO Set the state to CREKTING-KE_.
5OO If the response type of the current item iG 'Pree
,o ReGponse'.
- If nO key sets are defined for the item
yet.
- display the dialog graphically
55 depicted in figure A_q_(A)
85
EPO664O41B1
A-27
__o - accept and proce88 u8er input a8
_ollows.
5 - Exact _atch Radio B_t.ton
- aet _lag with c_rpent item
indicating that the examinee's _ree
1O _egponge mugt match the _ey e_a_tly
- contin_e at 55O.
- Rânge Radio _utton
,5 - set _lag with current item
indicating that the examinee's _ree
re8pon8e may _all into a nurneric
range
2O - continue at 55O.
- Cancel
- restore the matching t_e o_ the
25 current item to the value that
exi_ted prior to display o_ the
dialog
- continue at _OOO
3O - Key>__
- Save the key values with the item
- continue at 1OOO.
35 - Elge
- i_ range matching was de_ined _or
the cu_rent item via the dialog
depicted în A-4f(A)
4O - set atate to CREATI_G-_GE-_O_R.
- continue at _OOO.
1oOo Disable all _enu optiong by posting event
45 IP-GR__-8AR speci_ying all menu options are to be
disabled.
__oo Remove the dfalog dt_played in step 3OO.
__OO Redisplay the current item.
5O 13oo Proces8 interaction with the re8ponse while the
user identi_ies the keys. Continue until user
signals kev is complete via event KE_-CONP_ETE.
55 _ooO Retu_ to caller. 86
EPO664O41B1
A-2_
_-_G-_IT: .
5 A flowchart o_ this procedure is shown in Figures A-44
through A-48. .
loO _f Gtate is not CREATING-__, CREATING-_GE-_O_R,
or C_TING _GE UPPER
1O - -
- contin_e at llOO.
_oo Get the number of responses entered by the user.
If not, display error message and continue at l1OO.
15 3oo _ave the re_po_se info_ation with the current
item.
4oo If the response class is 'Pree Response'
2o - if the Gtate is CREATING KE_
- continue at 45O.
- if the state is CREATING-RANGE-UPPER
- set 6tate to CREATING KE_
25 _ . -
- COntinUe at 45O.
- if the state ie C_TING-_GE-_OMER
- clear the users response from the
3o GCreen
- continue at _OOD.
45o _f a response is defined, save the response.
5OO _et state to __ _TATE.
35 -
6oo Post event _P-UNGRE_-BAR epecifying all menu
options are to be enabled.
_oo _isplay the dialog graphically depicted in figure
4o A-46(A), updating the scrollable area of the dialog
with all key sets defined for the current item.
8oo Accept user input andprocess as follows.
- OK
45 - continue at _OOO.
- Delete All
- erase all key sets accumulated for
5O the current item
- continue at 8OO.
- View '
55 - highlight the response for the key
set selected in the scrollable area
87
EPO664O41B1
A-29
- continue at _OOO. .
- Delete
5 - delete the key set highlighted in
the scrollable area
- continue at 8OO.
1O _ Create
- pogt event M-KE__-_TE
- continue at tOOO.
,5 1ooo Remove the dialog displayed in step 7OO. _
11oo Return to caller.
M-__G-____
A _lowchart o_ this procedure ia ehown in Pigure A-Q9.
2O 1oo __ the item_e key ig merely a nu_er o_ r.egpongeg
(as speci_ied via _igure A-4O(A) during event
M-_CORE)
25 - digplay error mesgage
- continue at 1OOO.
_OO Display the dialog graphically depicted in _igure
A-49(A).
3O 3oo Accept uger input and p_ocege ag _ollowg..
- Cancel
- remove the dialog
35 - continue at _OOO.
4OO - Rrevious _
- _ighlight the response a_sociated
with the current key 8et (i.e. key
4O 8et __l)
- highlight the regponse associated
with the current key set
- continue at 3OO.
45 - Next
- unhighlight the response associated
with the current key set (i.e. key
5O get I_I)
- highlight the re8pon8e associated
with the current key set
- continue at 3OO.
55 88
EPO664O41B1
_-3O
lOOO Return to caller.
5 _-__ogg__ _
A _lowchart of this procedure is shown in Pigure A-5o.
1OO I_ the response type of the current item, as
1o defined vta figure A-5O(A) during event M-RE_Po__E,
iS nOt GRID, _TIPLE CHOICE, or ZONE_, continue at
lOOO.
,, 2OO Display the dialog graphically depict.ed in figure
A-qo(A).
3OO Accept ueer input and procesG as follows.
- OK
2O - check whether both the ,UGe Keys,
ent_ is undefined and 'number of
__ se_ections for key' is zero; if so,
25 di_splay err r d t_
O meSSage an COn inUe
at 3OO
- Save score t_e with current item
3o - remove dialog displayed in step 2oo
- continue at lOOO.
- Cance1
,, - Restore the score type of the current
item to the values that exieted prior to
step lOO.
- remove dialog displayed in step 2oo
4O - Continue at lOOO.
lOOO Return to caller.
45
5O
55 89
EPO664O41B1
_ci_lary Procedures A-3_ Check-Integrity-Procedure
_b__k __t_gritp-__o____r__ _
5 fl _ h t f th_ d _ h _ p. A __ t A
A oMc ar o is proce ure is S own in i_res - o -
52.
_oo If the current item ha8 no identi_ication number,
1o . add warning to error list,
_oo If the current item's. date/time of la8t
modification is not set or iG set to an invalidate
date/time, add warning to error list,
15 3oo If the current item has no stem component, the stem
file does not exist or is aasigned to an invalid
pane, add warning to error li8t.
2o qoo If there is a response component, and the response
file doee not exists or is assigned to an invalid
pane, add warning to error list.
_oo If there is a stimulus component, and the stimulus
25 file does not exist or iG asGigned to an invalid
pane. add warning to error list.
6oo If there is a directions component, and the
3o directions file does not exi8ts or is assigned to
an invalid pane, add warning to error list.
_oD If an invalid template t_e is cataloged for the
current item. or all panes of the selected template
35 are not used, add warning to error list.
8oo If there is a directions component and the
directions type is illegal, or there is no
4o directions component yet directions are assigned to
a specific pane, add warning to error list.
goo If the item's reaponae claas ia illegal, or no
response t_e i8 defined, or the response type is
45 illegal, or no parameters are defined for the
response, add warning to error list.
1ooo If any warning were added to the error liSt in the
5o above steps. and/or integrity checking was
specifically enabled via event M-INTEGRIT_, display
the error list. .
1_oo Erase the error list.
55 1_oo Return to caller. , ...
9O
EP O 664 O41 B1
APPENDIX B
Pseudo Code for the Item Preparation Tool
5 Main-Pro_edure
[O474] A flowchart of this procedure is shown in Figure B-1 .
1 OO Initialize the data base manager.
1O 2OO Load the names of all available packages.
3OO Perform event TP-MAl N -- subevent MAl N-l NIT (described bwlow) -- in the Packaging Manager.
35O Display the screen graphically depicted in figure B-1 (A).
39O Post event l DM-C-OPEN (described below).
4OO Fetch next event from event queue
15 5OO lf event is END-PRO_RAM
- continue at 8OO.
6OO Perform procedure associated with event.
2O _OO Continue at 4OO.
8OO Shut down the data base manager.
9OO Exit to operating system.
l DM-C-ABOUT
25 [O475] A flowchart of this procedure is shown in Figure B-2(A).
1 OO Display the screen graphically depicted in figure 1 .
2OO Process user input
3O - OK
- Remove the screen display shown in Figure B-2(C) from the screen.
- Continue at 3OO.
35 3OO Return to caller.
l DM-C-EXIT
4O [O476] A flowchart of this procedure is shown in Figure B-2(B).
1 OO Perform event TP-MAl N -- subevent MAl N-CLOSE -- in the Packaging Manager.
2OO Exit to the operating system.
45 l DM-C-NEW
[O477] A flowchart of this procedure is shown in Figure B-3.
1 OO Display the 'Create New Component' screen graphically depicted in figure B-3(A).
5O 2OO Fill the Packages field with the list of available packages.
3OO Fill the Type Code field with the list of allowable component types.
4OO Display the constructed Component l D for the default data.
5OO Process user input
55 - Package Select
- Display the list of available packages.
- Allow user to select a new package. 91
EP O 664 O41 B1
- Display constructed Component l D in the Component l D element.
- Continue at 5OO.
- Type Select
5 - Display the list of allowable component types.
- Allow user to select a new component type.
- Display constructed Component l D in the Component l D element.
- Continue at 5OO.
1O - Ok
- Continue at 6OO.
15 - Cancel
- Continue at 8OO.
6OO Perform event TP-MAl N -- subevent MAl N-CREATE (described below) -- in the Package Manager.
2O 7OO Perform Test-Preparation Procedure (described below).
8OO Return to caller.
l DM-C-OPEN
25 [O478] A flowchart of this procedure is shown in Figures B-4 and B-5.
1 OO Display the 'Open Component' screen graphically depicted in figure B-4(A).
2OO Fill the Packages element with the list of available packages.
3OO Fill the Components element with the component Id of all components in the package whose type matches the
3O currently selected type. If no type is selected, set type to PKG.
4OO Check the radio button matching the currently selected component type.
5OO Process user input
- Package Select
35 - Display the list of available packages.
- Allow user to select a new package.
- Fill the Components element with the component Id of all components in the package whose type match-
es the currently selected type.
4O - Continue at 5OO.
- Radio Button 'X' Select
- Change the current component type to match the type associated with the radio button.
45 - Fill the Components element with the component ld of all components in the package whose type match-
es the currently selected type.
- Continue at 5OO.
- Ok
5O 55O - lf no component ld is selected in the Components element
- Display an error message.
- Continue at 5OO.
- Continue at 6OO.
55 - Cancel
- Continue at 8OO. 92
EP O 664 O41 B1
- Component Select
- Continue at 55O.
5 6OO Perform event TP-MAl N -- subevent MAl N-OPEN -- in the Package Manager, now.
7OO If step 6OO failed
- post event l DM-C-OPE N.
- Continue at 8OO.
1O 75O Perform Test-Preparation Procedure.
8OO Return to caller.
l DM-C-SAVE
15 [O479] A flowchart of this procedure is shown in Figure B-6(A).
1 OO Perform event TP-MAl N -- subevent MAl N-SAVE -- in the Package Manager, now.
2OO Return to caller.
2O l DM-C-SAVEAS
[O48O] A flowchart of this procedure is shown in Figure B-6(B).
25 1 OO Pe_orm event TP-MAl N -- subevent MAl N-SAVEAS -- in the Package Manager, now.
2OO Return to caller.
l DM-E-UNDO
3O [O481] A flowchart of this procedure is shown in Figure B-6(C).
1 OO Perform event TP-MAl N-- subevent MAl N-U NDO -- in the Package Manager, now.
2OO Return to caller.
35 l DM-O-AU_O
[O482] A flowchart of this procedure is shown in Figure B-7(A).
1 OO Set the Auto/Manual flag to Auto.
4O 2OO Return to caller.
l DM-O-MAN
[O483] A flowchart of this procedure is shown in Figure B-7(B).
45 1 OO Set the Auto/Manual flag to Manual.
2OO Return to caller.
l DM-V-* (View-Option)
5O [O484] A flowchart of this procedure is shown in Figure B-7(C).
1 OO Perform event TP-MAl N -- subevent MAl N-Vl EW -- in the Package Manager, now, with a copy of the l DM-V-*
event as a parameter.
55 2OO Return to caller. 93
EP O 664 O41 B1
Pa_kage Manager
Event TP-MAl N
5 MAl N-C_O_E
[O485] A flowchart of this procedure is shown in Figure B-8.
1 OO Loop through each of the component types. For each type checkto determine if there is altered data present. If so,
1O - Perform event TP-MAl N -- subevent MAl N-SAVE -- for the component, now.
- Release the lock on the component.
2OO Return to caller.
15 MAl N-l NIT
[O486] A flowchart of this procedure is shown in Figure B-9.
2O 1 OO lnitialize all parameter and unit screens used in the application.
2OO Return to caller.
MAl N-CREATE
25 MAl N-OPEN
[O487] A flowchart of these procedures are shown in Figures 1 O(A) to 1 O(C).
1 OO Check to determine if there is already a component of the same type as the caller's parameter open. If so,
3O - Perform event TP-MAl N -- subevent MAl N-SAVE -- in the Package Manager, now.
- If the user canceled the save during processing of the MAl N-SAVE subevent
- continue at 1 OOO.
35 - Release the lock on the component.
3OO Hide the current Unit Screen display, if any.
4OO Initialize holding areas for the new component type about to be created/loaded.
4O 5OO lf the subevent is MAl N-CREATE
- Create the component in the database.
else
45 - Load the specified component from the database.
- Lock the component.
6OO Enable the component type of the specified component in the View option of the main menu.
5O 7OO Display the screen graphically depicted in figure B-1 O(D).
8OO Display the Unit Screen for the component type just created or opened in the Unit Screen area of figure B-1 O
(D). The Unit Screen displayed may be one of_. Scoring Unit Screen (figure B-1 O(E), Review Unit Screen (figure
B-1 O(F), Gl S Unit Screen (figure B-1 O(G), Break Unit Screen (figure B-1 O(H), Testing Unit Screen (figure B-1 O
(l), Tutorial Unit Screen (figure B-1 O(J).
55 1 OOO Return to caller. 94
EP O 664 O41 B1
MAl N-SAVE
[O488] A flowchart of this procedure is shown in Figure B-1 1 .
5 1 OO Display a query screen asking the user if he wants to save the component. The possible responses and courses
of action are as follows_,
- Yes
1O - Save the information of the previous component in the database.
- Set return indicator to 'saved'.
- Continue at 1 OOO.
- No
15 - Set return indicator to 'not saved'.
- Continue at 1 OOO.
- Cancel
2O - Set return indicator to 'canceled'.
- Continue at 1 OOO.
1 OOO Return saved/not-saved/canceled indicator to caller.
25 MAl N-SAVEAS
[O489] A flowchart of this procedure is shwon in Figure B-1 2(A).
3O 1 OO Display the 'Save As' screen graphically depicted in figure B-1 2(B).
2OO Allow user to select a Package and Type Code, and enter Name Code, Sequence Number, etc.
3OO Process user input_.
- Ok
35 - Create the component in the database using the package specified by the user via the 'Save As' screen
(ignore errors if the component already exists).
- Save the component information.
- Continue at 1 OOO.
4O - Cancel
- Continue at 1 OOO.
45 1 OOO Remove the screen displayed in step 1 OO.
1 1 OO Return to caller.
MAl N-Vl EW
5O [O49O] A flowchart of this procedure is shown in Figure B-1 3.
1 OO Hide the current Unit screen.
2OO Show the screen associated with the component specified by the caller's parameter.
3OO Return to caller.
55 MAl N-UNDO
[O491] A flowchart of this procedure is shown in Figure B-1 4.
95
EP O 664 O41 B1
1 OO Undo the last action performed against the current component.
2OO Return to caller.
Test-Preparation Pro_edure
5 [O492] A flowchart of this procedure is shown in Figures B-1 5(A) and B-1 5(B).
1 OO Process user input or menu events as follows_.
1O - Review
- If no component is selected in the Component List Area
- Continue at 1 OO.
15 - Else
- Display the Unit Screen (e.g. Scoring Unit, Review Unit, etc.) associated with the selected component
in the Unit Screen Area.
2O - Set screen values to the saved values of the component (non-editable).
- Continue at 1 OO.
- Del
25 - lf a component is selected in the Component List Area
- Delete the selected component from the Component List Area (not from the database).
- Continue at 1 OO.
3O - Open
- If a component is selected in the Component Select Area
35 - Perform event TP-MAl N -- subevent MAl N-OPEN -- in the Package Manager, now.
- Continue at 1 OO.
- Package Select (Select Area)
4O - Display list of available packages.
- Open package selected by user.
- Display list of components (from the Package) in the Select Area that match the current requested type.
- Continue at 1 OO.
45 - Element Select (Select Area)
- Copy selected element to Component List Area
- Continue at 1 OO.
5O - l DM-*
- Post event l DM-*
- Continue at 1 OOO.
1 OOO Return to caller.
55 96
EP O 664 O41 B1
APPENDIX C
Pseudo Code for the Test Pa_kaging Appli_ation
5 Main Pro_edure
[O493] A flowchart of this procedure is shown in Figure C-1 .
1 OO Display the screen (graphically depicted in figure C-1 (A) containing the four menu options and suboptions item-
1O ized below_.
- File
- Open (Menu-File-Open)
- Exit (Menu-Exit)
15 - Create Package (Menu-Create-Package)
- View Package (Menu-View-Package)
- Final Lock Package (Menu-Final-Lock-Package)
2OO Wait for user to choose menu option. Dispatch to the menu option procedure. The procedure for each menu
2O option is listed in parenthesis to the right of the option name, above.
3OO Continue at 2OO.
Menu-File-Open.
25 [O494] A flowchart of this prcedure is shown in Figure C-2(A).
1 OO Display the screen graphically depicted in figure C-3 to obtain the name of a BLOB from the user. The package
name is saved for use in Menu-View-Package.
2OO Return to caller.
3O Menu-E_it.
[O495] A flowchart of this procedure is shown in Figure C-2(B).
35 1 OO Exit to the operating system.
Menu-Create-Pa_kage
[O496] A flowchart of this procedure is shown in Figures C-4 through C-6.
4O 1 OO Display the screen graphically depicted in figure C-4(A) through which the user enters control information to
the process.
2OO Accept user data into the screen until one of three events occurs_. OK, CANCEL, or GENERATE is chosen. The
main data elements accepted from the user are as follows_.
45 - the name of an existing package rebuild or a new package to create
- the type of build, namely, draft, blue line, or final
- where the component data is located
- whether to encrypt the BLOB that's produced. Processing for each of the events is as follows_.
5O - OK_.
- Continue at 2OOO.
- CANCEL_.
55 - Continue at 2OOO.
- GENERATE_. 97
EP O 664 O41 B1
- Continue at 3OO.
3OO Initialize an empty working index of components.
4OO If in step 2OO the user chose to start the build at the Test Unit level_.
5 - Perform the Build-lndex-Procedure (described below) starting at the specified Test Unit.
- Continue at 9OO.
5OO If in step 2OO the user chose to start the build at the Session level_.
1O - Perform the Build-lndex-Procedure (described below) starting at the specified Session.
- Continue at 9OO.
6OO Add a reference to the package component to the working index.
15 7OO List each session referenced in the package.
8OO For each session referenced in the package, perform the Build-lndex-Procedure (described below).
9OO Sort the working index and remove duplicate component references.
1 OOO Add each component referenced in the index to the working BLOB.
1 1 OO Add the default Test Delivey Application Message to the BLOB.
2O 1 2OO Add the default Test Delivey Application Help Screens to the BLOB.
1 3OO Produce the ancillary files that are by-products of packaging, namely,
- the . PP file, which is used by the Administrative Application to list the available tests and spiral delivery
- the . PPS file, which is used by the Administrative Application to control spiraling
25 - the . PK_ file, which contains the component identifier of each Test Script in the BLOB
- the .ITB file, which contains the component identifier of each Item Table in the BLOB
- the .SKM file, which contains the Scoring Keys of each testable item in the BLOB.
1 4OO Save the BLOB, its associated index, and all ancillay files.
3O 2OOO Return to caller.
Build-lnde_-Pro_edure.
[O497] A flowchart of this procedure is shown in Figure C-7.
35 1 OO Add the specified component to the working index.
2OO For each referenced component in the specified component_.
- Perform the Build-lndex-Procedure for the referenced component.
4O 3OO Return to caller.
Menu-View-Pa_kage.
45 [O498] A flowchart of this procedure is shown in Figure C-8.
1 OO Load the BLOB specified by the user in Menu-File-Open. If none defined, display error and continue at 1 OOO.
2OO Display the screen graphically depicted in figure C-8(A).
3OO Read the index and display the component id, offset within the BLOB, size, level, tree, and last modification
5O date/time stamp of all index members_.
- sort the index by component id
- display the result in the scrollable area
- display information from the BLOB header, namely, the number of elements in the package, package type,
55 created by, and whether the BLOB is encrypted.
4OO Respond to user controls present on the screen as follows_.
98
EP O 664 O41 B1
- OK
- remove the screen from the display
- continue at 1 OOO.
5 - Sort List By_.
- display a list of the options by which the scrollable area can be sorted
- accept the user's choice
- sort the list according to the user's choice
- display the result in the scrollable area
1O - continue at 4OO.
- Limit List To_.
- display a list of all component types present in the BLOB, plus the default type ''ALL''
15 - accept users choice of component type (or ''ALL'')
- scan the BLOB for all components of the specified type and display components of the specified type in
the scrollable area
- continue at 4OO.
2O 1 OOO Return to caller.
Menu-Final-Lo_k-Pa_kage.
25 [O499] A flowchart of this procedure is shown in Figure C-9.
1 OO Obtain the package to lock from the user.
2OO Load the Package Index.
3OO For each component named in the Index, apply the final lock.
3O 4OO Copy the various files of the build to the Fl NAL area, namely, the BLOB, BLOB index, .PP, .PPS, and all . PKGs, .
ITBs, and .SKMs
5OO Display screen informing user that final locking is complete.
6OO Return to caller.
35 ____HD__ l3
_aa_do __d_ fo_ tba
4o _aat _a_i_a_ _pp_i__tio_
45
5O
55 99
EPO664O41B1
___-__o__d____ .
A f1owchart of this procedure is 8hown in Pigures D-_(A) and
5 D-_(B).
- _et the baae eystem state to _TATE-__.
- Link to the security file to obtafn permiseion to
1o run. If failure, return to operating system.
- Read parameter file passed from the adminiGtrative
application. Obtain the Examinee information (e.g.
,, name), teGt 8ession info_ation, and 'Start
Session' record for the Examinee Performance Record
(EPR) file.
- Initialize the system display. set colorG, etc.
2O - Initiali_e the subsystems of the delivery
application, namely,
- Review-Subsystem
25 _ Di_recti.ong-Subgygtem .
- Help-Sub8ystem _
- Break-Sub8ystem
3o - Load the normal and di8abled images of the
primary icons.
- Post event OSAM-SELECT -- subevent
3, SELECT-INITIUIZED.
- Petch the next event from the event _eue. Perform
the procedure aasociated with the event. Loop
until ,E_ PR__I event is retrieved from the
4O -
queue.
- Inform the Administrative Application of
termination and termination type.
45 _ _egtore the display and digplay colorg to the
condition that exi8ted at the start of procegsing.
- Return to the operating system.
5O
55 1OO
EPO664O41B1
O5AM-__Q. .
5_Q-5___.
, A flowchart of thiG procedure i8 shown in Pigure D-2(A).
_Oo _et the s_ate of the Remaining Time display to the
' 8pecified condition (on or off).
2OO Return to caller.
1O __Q-_I5P____ _
A flowchart of this procedure is ehown in Pi_re D-2(B).
_oo Display the title line. Include the remaining time
15 digplay if allowed for the current state.
2OO 'Return to caller.
5_Q-5T_T_5T.
2o A flowchaft of thi8 proCedUre iS ehOwn in Pi_reS D-3(A) and
D-3(C).
_Oo Initialize the default icons to be uSed in the test
from the Test Unit defaults.
25 2Oo _how the current icons.
3oo Broadcast the OSAM-STARTTEST event to all
Gubsystems.
3O qoo If not recovering a previouely failed geggion..
- If the current unit ie a UNIT-TESTUNIT
- Write a 'Start Test' record to the
3, eXaminee'g EPR file.
else
- _rite a 'Start EDC' record to the
examinee'_ EPR file.
4O 5OO If there are Teet Directions and they have not been
'SeenI.
- If recovering a previously failed session,
45 write a lRegtart _eggio_I record to the
examinee's EPR file. '
- Set the _system 8tate to STATE-DIÆ_IONS
,, - If recovering a previou81y failed eession and
a 'Start Directions' record has not been
written, or not recovering a session, write a
'_tart Directions' record to the examinee's
55 EPR file. 1O1
EPO664O41B1
- Indicate recovery is complete -
- Post event OSAM-DIRECT -- subevent
5 DIRECT-REQ_ST -- to the Direction_ Subsystem.
- Continue at _OOO.
_'Oo Ie recovering a previously eailed session.
- Post event OSAM S_Q -- subevent S_Q STAR__T
1O - - '
Event inc1udes a pa_amete_ indicating the
stapting Te8t Unit.
- Continue at lOOO.
15 _oo pogt event o__-SEQ __ 8ubevent _EQ_STAg_IT.
Event includes a parameter indicating start at the
Test Unit number l.
2o 1OOO Return to caller.
S__-G_AR__:
_ elowchart oe this procedure is ehown in Pi_res D-Q(_) and
D-4(E).
25 1OO If the current state is STKTE-INTEST.
- Ie the requested test unit is at or beyond the
number oe units in the Testing Unit.
3O _ set the bage gygte_ gtate to
STATE-_TEST
- post event OSAM-S_Q -- subevent
35 SEQ-_TEST
- continue at _OOO.
else
- Ie the requested se8gion unit is at or beyond
4O h _ e . . h .
t e nu er O unitS in t e Se_StOn.
- post event OSAM-5EQ -- subevent
SEQ-_GESSION
45 - ContinUe at 1OOO.
_OO Dete_ine type oe unit to be delivered. Procese
each type as follows.
- Re_lar Testing Unit'
5O - Set the ba8e system state to
STATE-I_ST.
- Post event O5AM-SEQ -- subevent
55 SEQ-STAg_ST __ epeCieying the uni.t
1O2
EPO664O41B1
numbe_. .
- Continue at lOOO.
5 - 5eCtion U_it
- Save the cu_rent system state and set the
new sy_tem etate to ST_TE-5TART5ECTION.
- Post event O5AM SEQ -- eubevent
1O -
5EQ-5TART5ECTION.
- Continue at _OOO.
- GfS Unit
15 - I_ recoveri_g a previougly _ai_led teeti_ng
SeGsiO_
- If the c_rrent sesgion unit is a
2o Tegting Unit or Zxaminee _ata
Collection Unit, write a 'Restart
Session' record to the examinee,B
EPR _ile.
25 - Write an 'End GIS' record to the
exarninee's EPR file.
- fndicate recove_ is complete.
3O - post event oS_-S_Q __ Gubevent
5EQ-START__T. Parameter indicates
advance to the next unit.
35 - COnti_Ue at _OOO.
- Save current system state and set new
state to STATE-GIS.
- _rite a '5tart GIS' record to the
4O examinee'g EPR _ile.
- Digplay the General In_formation Screen
speci_ied by the unit's confi_ration
45 data until dis_igged by uger.
- Write an 'End GIS'' record to the
' examinee's EPR file.
5o - ReStOre the Byste_ state tO itS preViOUS
v_lue.
- Post event OS_-SEQ -- subevent
SEQ STAR_IT. Parameter indicates
55 -
advance to the next unit.
1O3
EPO664O41B1
- Continue at _OOO. .
- Tutorial Unit
, - Post event OSAM-SEQ -- subevent
SEQ-RUN-MORIAL. Event includes
i_dicator specifying the tutorial to run.
- Conti_ue at _OOO.
1O - Break Unit
- If not performing recove_ of a previous
failed testing se_aion.
15 _ Save the current gy_te_ gtate and
set the new state to STATE-B.RE_.
- Post event OSAM-SEQ -- subevent
2o SEQ_B_.
- Continue at _OOO.
- Else
- __ the current sesaion unit is a
25 Testing Unit, write a 'Restart
Seaaion' record to the exa_inee's
EPR _i2e
3O - _f the tegting gegsion failed befo_e
the 'Znd Break' record was written,
write an 'End Break' record to the
3, examinee's EPR file.
- Indicate recovery is comp2ete.
- Post event OSAM-SEQ -- subevent
S_Q STAR_IT. Parameter indicates
4O d - h .
a Vance tO t e neXt Unit.
- Continue at _OOO.
- Score Reporting Vnit
4, - If _ot recovering from a previously
fafled teating sesaion, _ite a 'Start
_core Report_ _ecord to the examinee's
,o EPR file.
- Ca2culate the score for each scrollable
testing unit in the seaaion.
- Check the seaaion con_iguration
55 i_nfo_ati.on to determi.ne i.f ecoree are
1O4
EPO664O41B1
reported on ecreen. IEUR eo, display the
SCOres.
5 _ IEUR the gcore digplay ig te_inated by
administ_ative override, continue at
_OOD.
1o - Write an 'End _COre RepOrt' reCord to the
examinee's ZPR _il_.
- Po8t event O_AM-_EQ -- subevent
,, _EQ-_T__IT. Parameter indicates
advance to the next unit.
- Continue at _OOO.
_ooo Return to caller.
2O
25
3O
35
4O
45
5O
55 1O5
EPO664O41B1
_8Q-_TARTS8CT_O__
A _lowchart o_ thiG procedure is shown in Figures D-5(A) to
5 D-5(F3.
_OO Prepare the title line
2OO _et up the icone _or thia aection.
3OO I_ the calculator is available in this section,
1O initialize it.
4oo Enable required iconS and disable inapp_opriate
OneS.
15 5OO __ the _on_i_ is _ged in thig ge_tio_, change the
help in_ormation accordingl_.
6oO _oad the item table(a).
2o 7OO I_ not recovering a previou81y _ailed session and
the section is not adaptive.
- Write a '_tart Section' record to the
examinee's EPR _ile.
25 8OO Broadcaet the OSAM-STARTSECTION event to all
subsystems.
9OO I_ the section iG timed, prime the _imer, but do
3O not start ti_ing. _go, indicate whether eeconds
are to be ghown in the remaining time dieplay.
_OOO I_ recovering an adaptîve test _rom a previouSly
35 _ailed testing session, post event OS_-CXT --
subevent CKT-__.
_2OO I_ recovering a previously _ailed testing gession.
- I_ not recovering an item or the section
4O d
ende .
- End the current gtate 3nd resume the
previous state.
45 - PoSt event O_AM-SEQ -- subevent
SEQ-STAR_IT. '
- Continue at 3OOO.
5o - I_ the time limit is less than the last
thresho1d, poat event OSAM-TI_R -- aubevent
_IMER-_T_ZS_O_D.
- I_ this is an egsay section, per_o_ essay
55 recovery OperationS.
1O6
EPO664O41B1
- _rite a 'Restart Se8sion' recor.d to the
examinee's EPR _ile.
, l3OO If there are section directions and they haven't
been 'seen'.
- If there is a time limit for this 8ection. and
this is not an es_ay section, and timing is to
1O . d. . l
begin When seCtiOn ireCtiOnS are dlSp ayed,
start the count down timer.
- Save the current 8y8tem state and set the
15 state to STAT_-DI_C_IO_S.
- If recovering a previously fai1ed testing
session and a 'Start I7irections' record haG
2o not been written to the examinee's EPR file,
or a recovery is not in process, write a
'Start Directions' record to the EPR file.
- Indicate recovery is complete.
25 - Perfo_ event O.SAM-DIRECT -- s_event
DIRECT-RE_UEST -- now.
- continue at 3OOO.
3O lqoO _f this ie an adaptiye sectjon, continue at 3Doo.
_5Oo Change the 8ystem state to STATE-INSECTION.
l6OO If there is a time limit for this section, and this
3, is not an essay section, and timing is to begin
when section directions are diSplayed, start the
count down timer.
l7OO If not reco_ering from a pre_iou8ly failed testing
4O SeSSiOn.
- If displayed, hide and clear the calculator.
- Rrime the aystem to start at the first item.
45 - _ontinue at 3OOO.
l8OO (Zlse, recovering) If this 'i_ not an essay 8ection
and the aection has a time limit.
,o - Znable the Time icon.
- Indicate that the timer is not counting, yet.
_9OO If the last item did not end, redisplay the last
item displayed and continue at 3OOO.
55 '
2OOO If the ,last item ended via Next, perform the
1O7
EPO664O41B1
Move-To-Next procedure. .
Else, i_ the last item endedvia Prev, display the
, previous item.
Else, i_ the last item ended via Review.
- Save the current system state and set the new
state to _TATE _IEW.
1O - .
_ - _rite a 'Start_ReVieW' reCOrd to the
examinee's E_R _i_e.
- Post event OS_-REVIEW -- subevent
1, REVIEW-REQ_ST -- to the Review
5ubsystem.
- Continue at 3OOO.
,, Else, display the current item.
3ooo Return to caller.
_EQ-STARTIT__
A _lowchart o_ this procedure i8 shown in Pi_res D-6(A) to
25 D-6(C).
_OO I_ this is the eirat item in a group and there are
gxoup directions and the group direction8 have not
3o been 'Been'.
_ Save the current system 8tate and 8et the new
state to STAT_-DI_CTIO_5.
,, - Ie recovexing a previougly eailed testing
aeaaion and a '5tart Directions' record was
not written to the exa_inee'_ EPR eile, or
thi_ is not a reco_ery, write a 'Start
4O _i.rectiong, record to the EpR ei.le.
- Indicate recovery is complete.
- Post event OSAM-DIRECT -- subevent
4, DI_CT-REQUE5T -- to the Directions Subsystem.
- Continue at 2OOO. _
2oo Ie thie is the eirst item in a set.
- Ie the set directions have not been 'seen,.
5O - Ie thie is an adaptive test, set the
eirst item equal to the current item.
- Ie there are set direct.ions.
55 _ 5ave t_e current gygtem gtate and
1O8
EPO664O41B1
_et the new state to .
STATE-DIRECTIO_S. _
5 - I_ recovering a previously _ailed
testing ses8ion and a 'Start
Directions' record has not been
written to the examinee's EPR file,
1O or not PeCovering, write a 'Start
Directions' record to the EPR.
- Indicate recovery is complete.
15 _ post event oS__DIRECT __ subevent
DIRECT-RE__ST -- to the Directions
SubGystem.
_o - Continue at 2OOO.
- Else, indicate eet directions were
'SeenI.
_5 3Oo Adju_t icon_ and tit1e line to match the state.
4oo Replace the current sy8tem state with
STATE-DISP__ITEM.
_oo Post event OSAM-SEQ -- subevent SEQ-DISP__ITEM.
3O _ooo _eturn to caller.
G_Q-_IG____T_;
A _lowchart of this procedure is shown in Figures D-7(A3 and
35 D-7(B).
loo If directions are on screen.
- Perform event OSAM-STOP in the Directions
Subsystem, now.
4O - Adjust icons and title line to match the
state.
2oo If not recovering a previous1y failed testing
45 8egsion, or recovering an examinee quit, op
recovering and administrator quit.
- If this is an adaptive section, write a IStart
5o C_T Item' record to the examinee's _RR fi1e.
- else, write a 'Start Linear Item, record to
the examinee'8 EPR file.
3oo Display the current item. '
55 4OO Display the calculator, if it 8ho_ld be displayed.
1O9
EPO664O41B1
_5OO _et flags u_ed by the Review _ub8yst.em to indicate
the item ha8 been 'eeen'.
5 7OO __ this is an adaptive section, post event O_AM-CAT
-- subevent CAT-PRE_E_ECT -- to the Cat _ubeystem.
8oo E18e, i_ this is an eGsay se_tion.
,o - _top timing.
- _f _ecove_ing from a ppeviou81y failed teGting
se8sion. read in the up-to-the-minute respon_e
information.
15 - DiGable i_appropriate keys on the keyboard.
- Perform es8ay initialization.
- _ndicate recove_ is complete.
2O _ooo _ t t ll _
e Urn O Ca er.
25
3O
35
4O
45
5O
55 11O
EPO664O41B1
G_Q-_GZCTIOM_ -
A _lowcha_t o_ this procedure i8 shown in Figure D-8.
5 lOO Perfo_ the ClO8e-SeCtiOn prOCedure.
2OO Restore the previous system state.
3OO Post event OSAM-SEQ -- subevent SEQ-STAR_IT --
advancing to the next unit.
1O 4OO Return to caller.
G_Q-__GT.
A _lowchart o_ this procedure is shown in Fi_re D-9.
15 lOO __ the section has not been closed, per_orm the
Close-Section procedure.
2OO Hide the current icone.
2O 3oo I_ the current unit is. a Test Unit write an 'End
,
Test' record to the examinee's EPR _ile
QOO else, write an 'End EDC' record to the EPR _ile.
25 5OO Broadcast event OSAM-ENDTEST to all subsystems.
6OO Forget all saved states and set the system state to
STATE-_L. _
,, 7OO Post event OS_-SEQ -- subevent SEQ-ST_TUNIT --
advancing to the next unit. .
8OO Return to caller.
G_Q _G_8GIOM.
35 -
A _lowchart o_ this procedure is shown in Figure D-lo(A).
lOO Write an 'Znd Ses8ion' record to the examinee's EPR
_ile.
4O _,2OO I_ tutorials are running, end them.
3OO Post event END-PROG_.
4Oo Return to caller.
45 GZQ-GTO_TI_R:
A _lowchart o_ this procedure is shown in Figure D-lO(B).
lOO I_ a timer ie running, stop'it.
2OO Return to caller.
5O G_Q-GT_TT_R.
A _lowchart o_ this procedure is shown in Fi_re D-ll(A).
loo start an operating 8ystem timer.
55 2oo R t t ll
e.Urn O Ca er.
EPO664O41B1
_8Q-8_. .
A _lowchart o_ this procedure i8 ahown in Pigure D-tt(B).
5 tOO Noti_y the operating syStem to Send OGA-_-B_
i_ the special key combination (Administrative
' Override) is st_ck.
,, 2OO Per_orm event OGAM-B_ -- Subevent BR_-REQUEGT
-- in the Break GubSyStem -- _ow.
3OO Return to caller.
_8Q R_ _OR_.
15 - -
A _lowchart o_ thia procedure is shown in Pigure D-t2(A).
loO 5ave the current system atate and set the state to
GTATE-_ORI_. _
2O 2oo 5tart the Tutori.al Deli.ve_ program.
3OO Post event OGAM-GEQ -- subevent 5EQ-DRIVE-TUTORI_.
4oO Return to caller.
25 _8Q-_R__-_ORIU.
A _lowchart o_ this procedure is shown in Pigure D-l2(B).
lOO In_orm the Tutori'al Delive_ Rrogram o_ the next
3o tutorial to deliver.
2oo I_ not recovering a previoualy _ailed testing
session, or per_o_ing recove_ and the current
state is GTATE IN5ECTION, write a '5tart Tutorial'
35 - . .l
reCOrd tO the eXaminee'S EPR _l e.
3OO Indicate recovery iS complete.
QOO Return to caller.
4O
45
5O
55 112
EPO664O41B1
Og_-g__gCT. .
gg_gCT-__T_IZ_.
5 A _lowchart o_ this procedure iG 8hown in Pi_re D-_3.
_oo I_ recovering a previously _ailed te_ting session.
- I_ the last unit o_ the test was completed.
,, - Post event O5AM-5EQ -- subevent
5EQ-END5E55_ON.
- Continue at _OOO.
- I_ the unit recovered to is not a 'Tegt Unit'
15 nor an 'Examinee Data Collection Unit'.
- write a 'ReGtart 5es8ion' record to the
examinee's EPR _ile.
2O _ Elge
- 5et the current system state to
5TATE-I_Z5T.
25 - Pogt event O5AM-5EQ -- s_event
SEQ-5TARTTE5T.
- _oad system data associated with the
3o unit.
- Continue at _OOO.
- Post event O5_-5EQ -- subevent
SEQ 5T__IT -- indicating the recovered
35 -
unit.
- Continue at _OOO.
_oo Initialize the examinee's EPR _ile.
4O 3oo write a l5tart SeggionI record to the examineeIg
EPR _ile.
4oO Post event O5AM-5EQ -- s_event 5EQ-5TAR_IT --
45 indicating start at the _irgt unit.
_ooo Return to caller.
5O
55 113
EPO664O41B1
O__-_T. -
_T-__:
, _ _loWChart Oe thiS prOced_re iS ShO_ in Fi_re D-lQ.
lOO Ie not recovering a previo_sly eailed testing
session, continue at lOOO.
,, 2OO I_ this _ection is timed, start the remaining time
counter and enable the Time icon.
3OO _oad ineormation required to reGtart at the point
oe eailure erom the examinee's EPR record.
15 qoo I_ th t t, h _, l, ,t d t, ,
e CUrren SeC iOn aS a ime imt , an iming
needs to start beeore section directions are
displayed, and this is not an esSay section, start
,o t_e remaining ti_e counter and enable the _ime
iCOn.
5OO __ there are eection directions and they have not
,, been 'seen' by the examinee.
- Save the current System state and set the
system state to ST_TE-DIÆCTIONS.
- Per_orm event OSAM DIRECT -- subevent
3O -. h . ,
_IRECT-.ÆQUEST -- in t e DireCtton_ Subsystem,
nOW.
- Continue at _OOO.
3, 6OO Display the next item.
7OO Ie the testing aeasion _ailed while in Review, po_t
event ICON-_.
_OOO Return to caller.
4O _T-88__CT.
_ elowchart oe this procedure is shown in Fi_re D-l_.
_OO Write an 'End Item' re_ord to the ex_minee's EPR
45 ei.le,
2OO Ie there are no more items 'in the teat.
- Display the 'End oe Teat' mesBage to the
,o eXaminee.
- Change the current syatem state to
STATE-ENDSECTION.
- Post event O_AM SZQ -- subevent
55 -
SEQ-_SECTION.
114
EPO664O41B1
- Continue at lOOO.
3OO Dtsplay the next item.
5 _OO RetUrn tO Caller.
O_AM-_I__ .
1O _ _lowChart o_ thig proCedure ig gho_ in _igu_e D__6.
loo End _TATE-REVTEM. Pop the previo_s eystem state.
_oo Mrite an 'End Review' record to the examinee's EPR
15 _ile.
3oo T_ during Review a new item was selected (see
Review-_ubsy8tem.Review-_electt, diGplay the
,, epeci_ied item. Continue at 5OO,
qoo Else, re-display the item that was on screen when
Review was entered.
_oo T_ the calculator was displayed be_ore Review was
25 entered, re-di8p1ay it,
6oo I_ the state, which is restored in step 3OO, ie
_TATE ENDITEM, post event O_AM SEQ -- subevent
3O - -
SZQ-STARTITEM.
_oo Return to caller.
35
4O
45
5O
55 115
EPO664O41B1
O_AM-___; _
A _lowchart o_ this procedure is shown in Figure D-_7.
5 _oO _f subevent ig not H_kP_RE_, continue at 7oo.
2OO Perform event OSA-STOP in the Help-Subgystem, which
causes it to clean up from the moet recent
1o Help-ReqUest.
3OO __ the calculator waa on screen prior to Help,
redisplay it.
,, QoO _rite an 'End Help' EPR record to the examinee,s
EPR _ile.
5OO Restore the _ystem state that was in effect prior
to switching to STATE HE_P.
2O -
6oO Restore the display to the condition that exiGted
prior to switching to Help.
7oO Return to caller.
25
3O
35
4O
45
5O
55 116
EPO664O41B1
O__-_IR_CT.
A _lowchart o_ this procedure is shown in Pigures D-_8(A) and
5 D-_8(B).
lOO I_ subevent iG not DIRECT-RE_, continue at .7OO.
_oo Regtore the _ystem atate that waa in e__ect prior
,, to switching to 5TATE-DIRECTIOM5.
3oo write an 'End Directions' record to the examinee's
EPR _ile. _
Qoo I_ the current state is 5TATE I_E5T.
15 -
- mark the directions as 'seen'
- post event O5AM-5EQ -- subevent SEQ-5TARTUNIT
- continue at 7OO.
2O _oo __ the current gtate i_s STATE_5T_T5ECTIoM..
- mark the directions as 'seen'
- replace the current system state with
25 STATE-IM5ECTIOM
- start the section timer, i_ required
- display the _irst item o_ the section
3o - continue at 7OO.
6oo I_ the current state is 5TKTE-STARTITEM.
- I_ only Group DirectionG were displayed.
- mark them as 'seen'
35 - post event OS_-5EQ -- 8ubevent
5EQ-5TARTITZM
- continue at 7oO.
4O _ I_ Set Directiong were digplayed..
- mark them as seen .
- change the current atate to
45 STKTE-DI5P__ITEM
- post event O5_-SEQ -- subevent
SEQ-DI5P__I_M '
- continue at 7OO.
5O 7oo Return to caller.
O_AM-__.
55 117
EPO664O41B1
_ _lowcha_t o_ this procedure is shown in Figure D-_9.
_OO I_ s_event is not B_-RE_, continue at qoo.
5 2OO Re8tore the 8y8tem state that was in e__ect prior
to executing the Break.
3OO Post event O_AM-_EQ -- subevent _EQ-_TAR_IT.
,, 4OO Return to caller.
OGAM-CUC. .
_ _lo_chart o_ this procedure is shown in Figure D-2O.
15 b . _ __ R QuE__ .
_OO I_ sU eVent iS nOt CU _T _ E _ COntin_e at
3OO.
_oo Trane_er the ca_culator value to the reGponse.
2O 3oo Return to caller,
ogAM _ a_. .
25 _l- h- t _ th. ed _g gho_ i_n Fi._re D __
_ owc ar o iS prOC Ure i - .
.,,.,,.,,.,,T,hig mesSage is received directly _rom. admin oVerride
3o io'o'.' Di_play a query screen asking the administrator to
con_i_ his intention to end the break.
_oo I_ the administrator responds a__irmatively_
35 redirect special key Sequencee tO
osa-_dmin-Override.
3oo Remove the que_ Gcreen,
qoo Return to caller.
4O
45
5O
55 118
EPO664O41B1
T_' __ __t8 - O_AM-T__R
, O__-__.
.__g-___T.
A flowchart of this procedure is shown in Figures D-22(A) to
1o D_22(B). ,
_OO Ie the remaining time display i8 active, update the
remaining time display.
2OO Stop timing.
15 3oo If a message is currently displayed, clear it.
QOO If the current system state is STATE-HE_P, write an
'End Help' record to the examinee's EPR eile.
2O _oo Ie the current system state i,s STATE DIRE_TIo_S,
write an 'End Directions' record to -the examinee's
EPR eile.
2, 6OO If the current system state is STATE-DI5P__ITEM,
but Review is not. active.
- If this is an e88ay section, instruct the
,, eesay system to clean up and close down.
- Else.
- Hide the current item display
- Ie the current item was an8wered
35 previously.
. - Ie the answer was erased, display a
time out message.
4O _ Elge, dete_ine ie the previoug
answer matc-he8 the current answer.
If so, the item has not changed;
4, . display a time out message.
Othe_i8e, di8play a message
informing the examinee of the time
,, out condition and asking ie he wants
the previous or current answer.
Record specified answer.
- Else, the item was not previously
55 d
anSWere . 119
EPO664O41B1
T___ __ta ' O_AM-T_R
5 - If no angwer is currently selected,
display a time out message.
- El8e, display a message informing
,o the eXaminee o_f the time oUt .
condition and ask if he wants to
record the current answer. Record
,, an8wer accordingly.
- Write an 'End Item' record to the
examinee's ERR file.
7OO - If the curpent system state is STATE REVIEM;
2O -
- Perfo_ event OSAM-STOP in the
Review-Subsystem -- now.
- _rite an 'Znd Review' record to the
25 exami_nee,s EpR fi_le.
- End STATE-REVIEM and restore the system
state to what ie was before Review was
3o entered.
- Display a time out message.
8OO - Drop all saved system states and return
,, to the base state.
9OO - 5et the current syetem state to
. STATE-ENDSECTION.
95O - Post event OSAM SEQ -- eubevent
4O SEQ E_SE_IoN,-
lOOO - Ret-urn to caller.
___g-__8O__;
45 A flowchart of thi8 procedure ig sho_ in _igure D*23(A).
lOO - 5et the remaining time display to flash 6
times. '
,o 2OO _ POSt eVent OSAM_TIMER -- sUbeVent
TIMER-C_GED.
3Oo - Return to cal1er.
55 12O
EPO664O41B1
__e_ __ta - OG_-__g
5 __g-____.
A _lowchart o_ this ppocedu_e is _hown in Pigures _-23(B).
_oo - Update the pemaining time display in the title
,o line.
_oo - Return to caller.
15
2O
25
3O
35
4O
45
5O
55 121
EPO664O41B1
T__g_ ____ta - O__-__g
5 ICO_-CUC_
A flowchart of this procedure i8 shown in Figure D-2Q.
_OO If the calculator is not enabled for this section,
_ continue at 5OO.
1O ' '
2OO If the calculat_r is currently displayed, hide the
calculator and continue at 4OO.
3OO Elee, since the calculator is not currently
15 di8played, show the calculator.
4OO Write a 'Calcu1ator' record to the examinee's EPR
record.
2o 5OO Return to caller.
ICO_-CO_.
A flowchart of this procedure is sho_ in Pigure D-25(A).
_OO Enable all currently displaved icons
25 2OO Disable the Confirm icon.
3OO Perform the Move-To-Next-Procedure.
4oo Return to caller.
3O ICo_-______
A flowchart of this procedure i8 8hown in Figure D-25(B).
lOO _ignal the item's responge ma_ager to erase all
35 examinee ChoiCeS and reStore the item to it's
unanswered __tate,
2OO Return to caller.
ICO_ __IT.
4O -
A flowchart of this procedure is shown in Fi_res D-26(A) to
D-26(c).
lOO Disable all icons currently displayed.
45 2oo If a_ ite_ ig currently iji_played (gtate =
_TATE-DI_P__ITEM), perform the
Check-Re8po_se-Procedure. If a fail indicator i_
5o retUrned, cOntinUe at 2OOO.
3OO Hide the current screen display.
4OO Display a screen (fi_re 36(e)) containing a
message info_ing the examinee of the consequences
55 of exiting the test. The screen containe two
122
EPO664O41B1
____r ___ta - og_-__g
, choices. 'P_oceed' and 'Return to where vou were,.
The examinee mav select either option.
Additionally, the administrator may enter the
8pecial key sequence to produce an Administrative
1O _
Override. or a timeout may occur while the measage
is di_played. Process each of the choices/events
as _ollows.
15 _ ,Proceed,..
- Continue at 6OO.
- 'RetUrn...'.
2o - Re-enable the CUrrent icons.
- T7n-hide the current di8play.
- Continue at 2OOO.
- Timeout or Adminf8_rative Override
25 - COntinue at 2OOO.
6oo stop timing
_OO If the current 8tate is 5TXTE-DI_P__ITEM.
3O - If the current gectio_ ig an e8gay gection
- _ide the currently diaplayed icons
- 5ignal the essay handler to clean up and
3, close dom.
- I_ the essay section was te_inated via
Administrative Override, continue at
2OOO.
4O - wri.te an ,End Item, record to the
examinee's EPR file.
9OO If direct_on8 are currentlv df8p_ayed (state _
4, sTXT_-D__CTIONs). _
- Signal the Direction8-subsystem to clean up
and clo_e dom.
,o - Write an '_nd _irections' record to the
examinee's EPR _ile.
- Continue at __OO.
_Ooo If Review i8 currently di8played (state =
55 s_XTE-_IE_).. 123
EPO664O41B1
T_e_ _e_t_ - O5_-T_XB_
, - Signal the Review-5ub8y8tem to clean up and
ehut down.
- _rite an 'End Review' record to the examinee_s
EPR _ile.
1O __oo Set the application state to _TATE-ENDSECTION.
__5O Po8t event O5AM-_EQ -- subevent SEQ-ENDSECTION.
2OOO Return to caller.
15 __O_-__.
A _lowchart o_ thig procedure ie shown in Pigure D-27.
lOO Toggle the state o_ the countdown timer. __ the
2o remaining ti__ iS diSplayed, StOp diSplaying _t,
and vice versa.
2OO Post event OSA-TIMER -- subevent TIMER-_GE_.
3OO Return to caller.
25 _CO_-Q__T.
A _lowchart o_ this procedure is shown in PigureG D-28(A) to
D-28(C).
3O _oo Disable all currently digplayed jcong. '
3oO I_ an item is currently di8played (i.e. state =
5TATE-DISP__ITZM), obtain and save the examinee's
3, reSpOnGe.
4OO Hide the current display.
5OO Display a screen (_i_re 36(_)) in_orming the
examinee o_ the congequences o_ _itting the teet.
4O The screen contains two choices. 'Proceed' and
'Return to where yo_ were'. The examinee may
select either option. Additionally, the
45 adminigtrator may enter the gpecial key sequence to
produce an Administrative Override. or a timeout
may occur while the.message is displayed. Process
,, each o_ the choices/events a8 _ollow8.
- 'Return...'.
- Re-enable the current icons.
- Un-hide the current display.
55 - Continue at 2OOO.
124
EPO664O41B1
__' g_ ___g _ O_AX-___
5 - 'Proceed'.
- Continue at 6OO.
- Timeout or Adminigtrative Override
1o - Continue at 2OOO.
6oo Hide the currently diaplayed icon8.
_oo I_ the gcreen di8played in step 5OO wa8 ended by a
time out or adminietrative override, continue at
15 2OOO.
8oo I_ an item wa8 on screen at entry to this procedure
(state _ _TATZ-DI_P__ITEM).
2O _ I_ thi_g i_g an eggay gecti_on, gi__al the eggay
item handler to clean up and close down.
- Write an 'End Item' record to the examinee's
25 ZPR _ile.
- Continue at __OO.
9oo I_ directions were di8played (8tate _
,, _TATE-DIRECTION_3 at entrv to thi8 procedure.
- Per_orm event O_AM-_TOP in the
_irections-_ub8y8tem, which causes the
.sub8ystem to clean up and close down.
35 - Write an 'End Directions' record to the
examinee's ZPR _ile.
- Continue at __OO.
4O _ooo I_ Reyiew wag digplayed (8tate __ _TATE-REVIEW) upon
entry to thie procedure.
- Per_o_ event O_AM-_TOP in the
45 Review-_Ubsy8tem, which cauaes the subsygtem
to clean up and shut down.
- Write an 'End Review' record to the examinee's
EPR _ile.
5O . '
__oo _et the application 8tate to _TATE-_TE_T.
___o Poat event O_AM-_ZQ -- subevent _EQ-E_TE_T.
2Ooo Return to caller.
55 125
EPO664O41B1
__g_ _g_ta O__-___
, I_O_-_:
A flowchart of thie proCedure is shom in Figures D-29(A) and
D-29(8).
1Do Get and save the examinee's re_ponse to the
1O '
currently displayed item.
2oo Perfo_ the Check-Response-Procedure. If a fail
indicator is returned, continue at 8Oo.
15 3oo Hide the calculator if it,g ghowing.
qoo Write an 'End Item' _ecord to the Examinee's
Performance Record (EPR) file.
2o 45O Change'the appliCation state to STATE-_ITEM.
5oo _ave the application atate (5TATE-_NDITEM) and set
the new state to 5TATE-_IEW.
6oo Write a 'Start Review' record to the examinee's EPR
25 file.
_oo Post event OSAM-REVIEW -- subevent REVIEW-REQ_ST
-- to the Review Subsystem.
3O 8oo Return to ca11er.
ICO_-__
A flowchart of this procedure is shom in Pi_re _-3O.
,, 1oo Togg1e the appearance of the Mark icon. If the
'Marked' version wa6 di_play, change to 'Unmarked,,
and vice verga. Also toggle the 'Marked' state of
the item.
4O 2OO Return to cal1er.
ICO_-___:
A flowchart of this procedure is shom in Figure D-3_.
4, 1oo Write a '_tart Help' record to the EPR file.
_oO Hide the CalCulator, if it's current1y displayed.
3oo Prepare a parcel of information u8ed by the Help
,, procedure. Information inc1uded in the parcel is
as follows.
- Test level directions, if they've been seen by
the examinee.
55 _ If the exa_inee hag entered a gection, the
126
EPO664O41B1
__er __ta _ O_gX-____
, _OlloWiflg i__O_atiO_ iS al_O __Cl_ded in the
parcel.
- Group directions, i_ they e_i_t and __ave
been seen.
1O - 5et dire_tions, i_ they exist and have
been seen.
- Pop-up item directions, i_ they exist and
15 have been geen.
4OO Include in the parcel an indicator as to the what's
currently displayed. Indicators are as _ollowa.
2o - _irections, i_ test, section, group, set, or_
pop-up ftem directions are currently displayed
(state _ 5TKTE-E7IRECTIO__).
- Review, i_ the review scr_en is currently
25 displayed (state _ 5TPiTE-REVIEW).
- Item, î_ an item is currently displayed (_tate
_ 5TATE-DI5P__ITEMt.
3O 5oo _ave the previoug application gtate and 8et the new
state to STKTE-HE_P.
6OO Post event OSAM-HE_P -- aubevent ME_P-RZQUE_T -_ to
,, the Help Sub_ystem.
_OO Return to caller.
ICO_-__:
A _lowchart o_ thi_ procedure i8 shom in Figure _-32.
4O _oO I_ an item is not curre_tly diaplayed (8tate !_
5TXTE-DISP__IT_), co_tinue at 5OO.
2OO Per_orm the Check-ReBpo__e-Procedure. I_ a _ail
45 indicator i_ ret__ed, co_ti_ue at 5OO.
3oo Write an 'End Item' record to the EPR _ile.
4OO Display the previous item.
,o _oo Return to caller
I_O_-__
A _lowchart o_ this procedure ig _hown in Fi_re D-33.
_DO I_ an item is not currently dtsplayed (e_ate !=
55 5TKTE-DI5P__IT_), continue at 5OO.
127
EPO664O41B1
__er __t_ ' O__-T_tg
5 2oo Per_orm the Check-Response-Procedure. I_ a _ail
_ indicator is returned, continue at 5OO.
3OO I_ the 'Con_irm Answer' option is elected _or thiG
,, section, disable the NEXT icon and enable the
CO_IRM icon. _
QOO Per_o_ the Move-To-Next-Procedure.
5Oo Return to caller.
15
2O
25
3O
35
4O
45
5O
55 128
EPO664O41B1
T___ __t_ O__-___g
, R__t_W _Ub__at_
_I_-_Q__T:
Zt _lowchart o_ this procedure is sho_ in Pi_re _-34.
_OO Prepare a screen _or display a8 per figure 4O. The
1O '
screen i8 hidde_ while it's being built.
2OO _oop through the li_t of items in the section,
examine the item, and emit lines to the scrollable
15 a_ea a_ followg..
- I_ the item belongs to a different group than
the previo_s item and the group has a
2o paraphrase, add the group paraphrase to the
scrol1able area.
- I_ the item belongs to a different set than
the previous item and the eet has a
25 paraphrase, _add the set parap_ase to the
scrollable area.
- Construct a line describing the item and add
3O same to the gcro1lable area. _tem
descriptionB contain the following.
- item number within the section
,, - the item's paraphraee, if any
- an indicator as to whether the item is
'an8wered', 'not answered', or 'not seen,
- an indicator as to whether the item is
4O 'marked'.
3OO _utomatically _croll up the list so the current
item is disp1ayed in the middle of the gcrol1able
45 regi'on. _
4OO _isplay an indicator at the top of the list to
indicate the relative position of the liet within
,, the scrollable area. Rules for di8playing the
indicator are as fol1ows.
- If the first line in the li8t is on screen,
display the 'hginning' indicator.
55 - If the lagt li.ne of the li.et i.e on ecreen_
129
EPO664O41B1
Tf_er 8ve_ta _ OGAx-T__
5 display the Ie_dI indicator.
- Otherwise, display the 'middle' indicator.
5OO _how the screen.
1o 6OO RetUrn to caller. _
OGAM-GTO_;
A _lowchart o_ this procedure is shown in Figure _-35(A).
15 A _lowchart o_ this procedure is shown in Figure D-3Q.
_OO Erase the Review display.
2OO Hide the Review display Gcreen.
2O 3oo R t t ll
e Urn O Ca er.
_CROLL-REQUE_T.
A _lowchart o_ this procedure is shown in Figure D-35(B).
25 _OO ReGpond to controls to move the list up and do_
through the scrollable area. Maintain relative
position indicator as described in step
3o Review-Request_4OO.
2OO Return to caller.
G___CT-_Q_GT:
A _lowchart o_ this procedure is shown in Figure D-36(A).
35 _OO De-select any previously_selected line.
2OO Highlight the line.
3OO Return to caller.
4O _oTo-g_o_..
A _lowchart o_ this procedure is shown in Figure D-36(B).
_OO Obtain the item number _rom the highlighted line in
45 the scrollable area. I_ a group or set paraphraGe
is highlighted, use the _irst item in the group or
set.
2OO PoSt eVent O_AM REVIE_ -- speci_y the item number
5O -
to display
3OO Return to caller.
55 13O
EPO664O41B1
T_' er __t_ O__-T__
5 __-__O_. .
_ _lowchart o_ this procedure is _hown in Pigure D-37.
lOO Post event O_AM-REV__ -- speci_y that the item
,, displayed when Review-Re_est was initiated is to
be re-displayed.
2OO Retu_n to ca_ler.
15
2O
25
3O
35
4O
45
5O
55 131
EPO664O41B1
__g_ 8v__8_ ' OS_-__R
, 8a__ S_ap_t_
___-_Q___.
A _lowchart o_ this procedure is shown in Figuree D-38(A) to
,, D-38(B).
_oo Prepare a screen for display as per figure 37. The
_creen is hidden while it's being built.
_oo Enable all buttons 8hown in figure 37,
15 3oo Tf Tegt Di._ecti.ong have no_ been .gee_, yet or they
do not exist, disable the Genera1 _irections
button.
,o 4oo Tf _ection Directions have not been 'seen' yet or
they do not exist, diGab1e the _ection Dire_tio_s
button.
,, _oo Tf Group, _et or Pop up _tem Directions have not
been ,seen' yet or they do not exist, disable the
_uestion Directions button.
6oo Tf this is not an essay section, hide the _ing
3O Di.recti.ons button.
7oo Determine which opening (or welcome) mesaage to
diBplay. The determination ie made as follows.
3, - If TeSt, _eCtiOn, GrOUp, _et Or pOp Up Item
Directions are currently on screen, a canned
welcome message _s 6elected. Continue at 8OO.
- If Review is currently on screen, a different
4O d h. h , , . .
Ca_e meSaage, W iC iS COnteXt Sen_itkVe, iS
selected. Contin_e at 8OO.
- If.an item is. currently on s_reen, one o_ the
45 following ig geleCted.'
- The item's Group, _et and pop up _tem
Directions are concatenated into a single
,, message. Tf the resulting mes8age ie _t
empty. becau8e one or more of these exist
for the item, continue at 8OO.
- If _ection DireGtion8 exist, the message
55 i.s get to the contents thereof, and the
132
EPO664O41B1
__e_ ____g _ O8AM-_I_R
, _ection Directions button ie disabled;
continue at 8OO.
- I_ Test Directions exist, the message is
set to the contents thereof, and the
1O General Direction8 button is disabled;
continue at 8OO.
- If none of the above exist. the message
15 .
iS Set to a Canned _eSSage; COntinUe at
8OO.
8oo The ;message' conetructed in step 7OO is Written to
2O the gcrollable region of the ecreen.
goo The gcreen is displayed,
_ooo Return to caller.
2, O__-g_O__
A flowchart of this procedure iS shown in Pi_re D-39(A).
_oo Restore the help display to a cleared condition.
_oo Hide the help digplay screen.
3O 3oo Return to caller.
gg_-B__
A flowchart of this procedure is shown in Pi_re D-39(B).
35 p t t o__ HE_p ub t HE_p RE_
_OO og eVen - -- S eVen - .
_oo Return to caller.
Q_gTIo_-_I_CTIO_G-__O__
4o A flowchart of this procedure ig shown in Pigure D-39(C).
_oo Disable the Question Directions button. Enable
whatever button waa previou_ly digabled.
,, _oo Concatenate any of the Group, _et or pop Up Item
directions available for th_e current item and
digplay 8ame in the scrollab1e area of the screen.
3oo Re_turn to caller.
5O g__Io_-_I_CTIO_g-B_:
A flowchart of this procedure is shown in PigUre D-4O(A).
_oo Digable the _ection Directions button. Enable
55 w_tever button w_g previougly digabled.
133
EPO664O41B1
____ ___g _ O__-T_R
5 _oo _isplay the _ection Directions in the scpollable
area of the acpeen.
3oo Return to caller.
1O
15
2O
25
3O
35
4O
45
5O
55 134
EPO664O41B1
T_g_ __ta O8_-T_g
, __-___CT_o_g-8__..
A _lowchart o_ thi8 proced_re ie ehown in Pigure D-4O(B).
_oo Diaable the General Direction_ button. Znable
1o whatever button was previouely dieabled.
2OO Diaplay the Teat Directions in the scrollable area
o_ the screen.
,, 3Oo Return to caller.
8OW-TO-8CRO__-8_O_.
A _lowchart o_ thi8 procedure is ehown in Pigure D-4l(A).
_OO Disable the How-to-Scroll button. Enable whatever
2O button was previously dieabled.
2oO Display the canned Scrolling Directions in the
scrollable area o_ the screen.
25 3oo Return to caller.
_I__-8_O__
A _lowchart o_ thia procedure is ahown in Pigure D-4l(B).
3o lOO Disable the _ing button. Enable Whatever button
was previously diaabled.
2oo Disable the canned T_ing Directions in the
scrollable area o_ the screen.
35 3OO Return to caller.
T_8TI_-TOO_8-8_O_;
A _lowchart o_ this procedure ia Ghown in Pigure D-42(A).
4O _oo Di_gable the Tegti_ng Toola button. Enable whatever
button was previously disabled.
2oO Display the Testing Toola screen as per _igure 38.
4, Teating Tools not enabled in thiS Section afe
elided prior to di8play. '
3oo Return to caller.
5O
55 135
EPO664O41B1
__e_ _a__g O5AM-____
5 __g____-_OO_ _8_.
A flowchart of this procedure is shown in Figure D-42(B).
_Oo _oad and di_pla_ the information file associated
,, with the chosen Teeting Tool button.
_oo Return to caller. '
15
2O
25
3O
35
4O
45
5O
55 136
EPO664O41B1
__e_ __ta _ O__-T_R
5 8____ _O_p_t_
O__-8_;
8__-_Q__T:
A flowchart of this proced_re i_ Ghown,in Pigures I7-43(A) to
1O D-43(_).
2oo _tart a co_tdown timer. Period is specified by
the caller.
15 _oo _iGplay a screen asking the examinee if he wants to
take a break. Accept the examinee's respon8e,
There are four events that may occur at this time.
2o 3oo - Examinee Declines the Break.
- _ost event OSAM-BR_ -- subevent
BR_-RE_.
- Continue at 2OOO.
25 .d h _
qoo - Administrator Overri es t e Brea .
- Continue at _OOO.
5oo - Message Times Out,
3o - _et a 5 second timer.
- _isplay a 'No ReGponse' screen. There
are two events that ma_ occur at this
time.
35 _ - Me8sage TimeG Out.
- Contin_e at 6OO.
- Ad_iniatrator Overrides the Break.
4O _ Conti.nue at _ooo,
6oo - Examinee Accepts the Break.
- _r.ite a 'Start Break' record to the
45 exa_inee's EPR file.
- Set a 5 second timer.
- _isplay the 'On Break' screen. There are
two events that may occur at this time.
5O - Timer Eqires.
- Contin_e at 7OO. _
- Administrator Overrides the Break.
55 - Continue at _OOO.
137
EPO664O41B1
__e_ _g_ta _ OG_-___g
5 7OO 5et timer to co_nt down the break period. There
' are two events that may occur at this time.
- Timer Ticks.
1o - Decrement remaining time. ProceGs
remaining time display as _ollowS.
- Time RemainG.
- Update the remaining
15 ti.me di.gplay.
- Continue at _OOO.
- No Time Remains.
2O - Clear the remaining time di_play.
- Set indicator that the break is in
overtime.
25 - Continue at _OOO. __'.
- In Overtime.
- _ncrement overtime count.
- Continue at _DOO.
3O .
- Admi'niStratOr End8 the Break.
- Continue at 8OO.
8oO Display the 'End Break' con_irmation screen. There
35 are three events that may o_cur at t_ig time..
- Timer Tick.
- ProcesG as per 'Timer Tick' in etep 7OO.
4o - Examinee Con_irms Screen.
- 5top timer.
- Calculate overtime, i_ any.
- Write 'End Break' record to
45 eXamineeIS EPR _ile. RecOrd
overtime.
- _oet event O5AM-B_ -- subevent
5o B_-RE_.
- Continue at _OOO.
- Adminietrator Overrides.
55 - Stop timer.
- Calculate overtime, i_ any.
138
EPO664O41B1
T___ ___t_ ' OG_-TI_g
5 - _rite 'End Break' record to examinee_g
_ EPR _ile. Record overtime.
- Continue at _OOO.
,o _OOO Return to. caller.
O5AM-GTO_: -
A _lowchart o_ this procedure is shown in Pigure _-44.
1oo I_ a Break SubByetem message is currentl_
15 dieplayed, remove it _rom the screen.
2OO Return to caller.
2O
25
3O
35
4O
45
5O
55 139
EPO664O41B1
T____ ___t_ _ O5AM-TI__
5 __t__ti___ 5Oayat_
OG_-_I_CT;
_IR8CT-_Q_GT_
A _lowchart o_ thi.s procedure ie _hown in Figure D-45(A).
1O _ _ h
1OO Check the parce or t e next directions Yile to
display. IY none.
- Post event OSAM-DIRECT -- subevent
15 D__EC_-RE_.
- Continue at 4OO.
2OO Display a Direction_ screen as shown in Yigure 35.
,, 3OO _oad the next directions Yile speciYied in the
parcel into the scrollable area oY the screen.
4OO Return to caller.
OGAM _TO_:
25 ' -
A Ylowchart o_ this procedure is shown in Pigure D-45(B).
lOO Terminate directions display, iY any.
3OO Return to caller.
3O GCRO_1i-_Q_GT.
A Ylowchart oY this procedure is shown in Figure D-Q6.
lOO Respond to controls to move the directions up and
35 do_ in the scrollable area. Maintain relative
position indicator.
2oO Return to caller.
__ __O_:
4O Yl -h Y h, d . h ,
A owc art o t iS proce ure is s own in Figure D-47.
1OO IY the return button has already been 'pressed' Yor
the current directions Yile.
45 - Dismiss current directï'onG Yile.
- Advance to next Yile in 1ist.
- Post event OSAM-__R_CT -- aubevent
DIRECT REQUEST.
5O -
- Continue at lOOO.
Else
- IY the bottommost portion oY the directions
55 has been di_played..
14O
EPO664O41B1
T_g_ _e_t_ _ O_AM-T_R
, - post event O_AM-DIR_CT -- SUbevent
DIRZCT-REQUE_T.
- Continue at _OOO.
,o - Else
- Indicate that the return button hae been
pressed _or the current directions _ile. '
- _isplay an error message in_orming the
15 examinee that directions remain to be
Seen.
- Continue at _oOO.
2O _ooo Return to caller.
Tf_er __ta _ O__ __g
25 -
Cbe__-Raa_o__e-__oGedure_
A _lowchart o_ this procedure is shown in Figure D-48.
3o _OO Get and save the examinee's cUrrent regponge.
2OO Check _or and react to the _ollowing conditions.
- Incomplete Response. I_ the examinee has
,, supplied any response, but the responee is
nonetheless incomplete, and eqlicit prompting
is speci_ied _or this eection. display an
error message and continue at 4OO.
4O - No Response. I_ the examinee has supplied no
response and 'Muet Answer' is speci_ied _or
thiG section, display an error message and
45 . oo
COntinue at Q .
3OO Return success indicator.
4Oo Return _ai1 indicator.
5O
55 141
EPO664O41B1
__g_ ___t_ . _ O__-T__R
, Mova-To-_ext-P_o_ad_r__
A _lowchart o_ this procedure is shown in Pigures D-49(_) and
D-49(B).
,, _oO I_ this section is .computer adaptive and the next
item is ready _or di8play, continue at 2OOO.
_oo I_ this section is not linear, continue at _OOO.
3oo __ the caller indicates the previous item is ended,
15 . d , d h .
Wrlte an 'En Item reCOr tO t e eXaminee'S EPR
_ile.
Qoo I_ there are remaining unseen items in this
,o section, erime the syetem to display the next item
and continue at aOOO.
5oo Hide the current item display.
,, 6oo Examine the section con_iguration in_ormation to
determine i_ REV_E_ ie permitted _or this eection.
Proces8 as _ollows.
- Review permitted. Display a screen (_igure
3O 36(d)) i.n_o_i.ng the exami.nee that there are
no more items. The screen provides three
choices. 'Return to where you were',
3, ,Proceed', or 'Go to Review'.
- Review not pe_itted. Dieplay a screen
(_igure 36(d) minus 'Review' option) in_orming
4o the examinee that there are no more items.
The screen provides two choices. 'Return to
where you were', and 'Proceed'.
_oo At this time 5 events are poasible. either the
45 exami.nee choo8es one o_ the. opti.ong on the gcreen
,
or a timeout or Administrative Override occurs
while the message displayed in step 5OO is on
,o Gcreen. Process the po8sible choices/events as
_ollows.
- 'Return...'.
,, - Re-display the current item and continue
,at 2OOO. 142
EPO664O41B1
_f_e_ ____a O_AM-___
5 - 'Go to Review'.
- Sa_e the current state and eet the
' new 8tate to STATE-REVIEW.
_ - Write a 'Start Review' record to the'
1O examinee'8 EPR _ile.
- Post event OSAM-REVIEW -- 8ubevent
REVIEW-REQUEST -- to the Review
15 Subgygtem.
- Continue at 2OOO
- 'Proceed'.
_o - _ndicate that the te_ination type ig via
NEXT.
- Porget any saved gygtem stateG. Set the
_5 new state to STATE-_SECTION.
- Post event OSAM-SEQ -- subevent
SEQ-ENDSECTION.
- Continue at 2OOO.
3O - Timeout or Administrative Override
- Continue at 2OOO.
_ooO Score this item (adaptive only)
35 2ooo Return to caller.
4O
45
5O
55 143
EP O 664 O41 B1
T_g_ ___t_ - OgAX-TI__
5 _loge-g__tfo_-__o_gd__g
A _lowchart o_ this procedu_e is _hown in Pigure D- 5 O ,
_ o o I _ thiG is an essay sectiOn '
1o - _hutdown the es_ay subsystem
_ o o Hide and clear the calculator .
3 o o _top t iming and clear the remaining time display ,
,, q o o I _ this is an adaptive 8eCtion
- compute the score
_ o o Write , End Section ' record to the e_aminee ' s EPR
_ ile .
2O 6 o o Broadcast event O_AM-_SECTION to all subsystem_ .
_ o o I _ this is an adaptive section , shut down the
adapt ive 8_system .
25 8 o o _hutdo_ the calculator .
_ o o o Return to caller ,
3O __ai_g
1 . A computer based testing system comprising_.
a test development system (1 O) for creating a computerised test having a number of questions to be answered
35 by an examinee and rules for determining a sequence of questions to be presented;
a workstation (3) operable to present the questions to at least one said examinee and operable to accept
examinee responses to the questions so presented; and
a test delivery system (1 2) operatively coupled to both said test development system and said workstation for
delivering said computerised test by presenting the questions according to the rules to each said examinee
4O on said workstation_,
characterised in that
said computerized test has a plurality of related test screens containing messages and directions providing
45 information to the examinee_, and by
a computerized test script (72O) created using said test development system (1 O) for controlling said worksta-
tion (3) to present the questions and related test screens according to rules defined by said test script (72O).
2. The system of claim 1 , further comprising_.
5O a test administration system (1 4) operatively coupled to said test development system, said test delivery sys-
tem and said workstation for initiating and terminating the delivey of said computerized test by said test delivery
system.
55 3. The system of claim 2, wherein said test administration system fu_her comprises security means for securing said
computerized test from access by unauthorized persons.
4. The system of claim 1 , wherein said test development system further comprises_.
1 44
EP O 664 O41 B1
a test document creation means (62) for defining the questions to be presented by the examinee based on a
test specification (686) and for determining a manner of presentation of the questions on said workstation
during the delivey of said computerized test;
an item preparation system (6O) interfaced with said test document creation means for computerizing the
5 questions for delivery by said test delivey system;
a test preparation system (57) interfaced with said test document creation means and said item preparation
system for preparing the computerized test script and the related test screens; and
a test packaging utility (58) interfaced with said item and test preparation systems for combining the compu-
terized questions, the test script and the related test screens to form the computerized test.
1O 5. The system of claim 4, wherein the test specification specifies a number of test sections of the computerized test
in which the questions are assigned, each test script defining the rules for determining the sequence of questions
and related test screens to be presented in one of said test sections and wherein said test preparation system is
operative to create, and the test delivery system is operative to deliver, at least one of_.
15 tutorials (71 4) for providing instructions tothe examinee for interacting with said workstation during the delivery
of said computerized test;
directions (71 O) for explaining how to take said computerized test;
a help facility (? HELP) selectable by the examinee for accessing additional instructions describing how to
2O interact with said workstation andforaccessing said directions for explaining howtotake the computerizedtest_,
a review (REVIEW) facility selectable by the examinee during the delivey of said computerized test for pro-
viding a list of substantially all of the questions of said computerized test and for allowing the examinee to
select a specific questions from the list to be presented; and
said test preparation system creating a session script (71 8) defining a sequence in which the tutorials, the
25 directions, and the test scripts are to be invoked by the test delivery system and wherein the test delivey
system is operable to interrupt the sequences defined by the session script and the sequences determined
based on the rules defined by the test scripts to invoke the help and review facilities upon selection by the
eXamlnee.
3O 6. The system of claim 1 , wherein a plurality of predetermined events are monitored by said test delivey system,
each event and related information being stored within a log record, a combination of log records created during
the delivey of said computerized test defining a performance file.
7. The system of claim 6, wherein said events include said examinee responses, the system further comprising_.
35 a data distribution system (1 8) having an interface with said test delivery system for formatting said perform-
ance files so that said examinee responses can be evaluated.
8. The system of claim 2, wherein a plurality of events indicative of errors of said computer based testing system and
4O related to security of said computerized test and said computer based testing system are monitored by said test
administration system, and a plurality of predetermined events occurring during the delivey of said computerized
test are monitored by said test delivery system, each event and related information being stored within a log record,
a combination of said log records storing events related to errors defining an error log file, a combination of said
Iog records storing events related to the security of said computerized tests and computer based testsing system
45 defining a security log file, and a combination of said log records storing events occurring during the delivery of
said computerized test defining an examinee performance file.
9. The system of claim 8, wherein the workstations are located at at least one test center (2), the system further
comprising_.
5O a data distribution system having an interface with said test administration system for processing said error
Iog files, said security log files and said examinee performance files and for generating from these files reports
indicative of events occurring at each workstation and at each of said test centers.
55 1 O. The system of claim 1 , wherein said computerized test is a standardized test.
11 . The system of claim 1 , wherein said workstations are located at at least one test center, the system further com-
prising_. 145
EP O 664 O41 B1
a data distribution system located at a central processing site (1 ) remote from said test centers, said data
distribution system having a data connection with each said test center for receiving information related to
activities at the test centers and information indicative of said examinee responses and for processing said
information to generate at least one report presenting said information in human readable form and for for-
5 matting at least some of said information for subsequent scoring of examinee responses.
12. A method of computer based testing, comprising the steps of_.
producing a computerized test having a plurality of questions to be answered by an examinee and rules for
1O determining a sequence of questions to be presented;
delivering said computerized test to the examinee;
recording responses provided by the examinee to the questions presented; characterised in that
said computerized test has a plurality of related test screens containing messages and directions providing
information to the examinee; and characterised by
15 a computerized test script (72O), created using a test development system (1 O) defining rules for determining
a sequence of the questions and related test screens to be presented; and in that said test delivery is accom-
plished by presenting said plurality of questions and related test screens to the examinee at a computer work-
station (3) according to the rules defined by said test script (72O).
2O 13. The method of claim 1 2, fu_her comprising the step of_.
administering at least one said computerized test to at least one examinee.
14. The method claim 1 2, further comprising the step of_.
25 securing said computerized test and responses from access by an unauthorized person.
15. The method of claim 1 2, wherein said computerized test is standardized and wherein said computerized test is
based on a test specification (686) being defined independent of said test development system used to produce
3O the computerized test.
16. The method of claim 1 5, wherein the test specification specifies a number of test sections of the computerized test
in which the questions are assigned, each test script defining rules for determining the sequence of questions and
related test screens to be presented in one of said test sections, wherein said step of delivering said computerized
35 test further comprises at least one of the following steps_.
displaying tutorials to the examinee for instructing the examinee how to interact with said workstation;
displaying directions related to the questions to explain how the examinee is intended to respond to each
question presented;
4O providing a help facility selectable by the examinee to access instructions describing how to interact with said
workstation and for accessing directions describing how to respond to questions presented; and
providing a review facility selectable by the examinee to review questions presented by providing a list of
substantially all questions included in said computerized test and for allowing the examinee to return and to
advance to a question listed; and
45 said method further comprising the steps of_.
creating a session script defining a sequence in which the tutorials, the directions, and the test scripts are to
be invoked_, and
interrupting the sequence defined by the session script and the sequence determined based on the rules
defined by the test scripts to invoke the help and review facilities upon selection by the examinee.
5O 17. The method of claim 1 2, wherein said step of delivering said computerized test further comprises the step of_.
providing user controls selectable by the examinee for controlling said delivery of said computerized test.
55 18. The method of claim 1 2, wherein said workstation comprises a memory, the method further comprising the steps of_.
registering at least some examinees to take a specified computerized test prior to the delivery of the compu-
terized test, each examinee so registered being authorized to take said specified computerized test;
146
EP O 664 O41 B1
storing at least one specified computerized test in said memory prior to delivery of said specified computerized
test.
1 9. The method of claim 1 8, further comprising the step of_.
5 safeguarding each computerized test stored in said memory from access by unauthorized persons.
2O. The method of claim 1 8, further comprising the step of_.
1O confirming the authorization of each examinee seeking to take said specified computerized test prior to deliv-
ering said specified computerized test.
21 . The method of claim 1 8, further comprising the steps of_.
15 selecting one specified computerized test stored in memory to be delivered, the test selected being said spec-
ified computerized test the examinee is authorized to take; and
initiating the delivery of said computerized test so selected.
22. The method of claim 1 2, wherein each said workstation is located at a test center.
2O 23. The method of claim 22, further comprising the steps of_.
transferring information indicative of activities occurring at each said test center from said test centers to a
central processing site being remote from said test centers; and
25 processing said transferred information so that said activities at said test centers are monitored at said central
processing site.
24. The method of claim 22, further comprising the steps of_.
3O transferring said recorded responses from each workstation to a central processing site being remote from
said test centers_, and
formatting said recorded responses for subsequent scoring.
25. The method of claim 1 2, wherein the step of producing a computerized test comprises the steps of_.
35 preparing a test document of items, each item having at least a test question, related presentation information
indicative of at least the text to be presented to an examinee and a response type associated with said item;
preparing an electronic form of each item according to said presentation information and said response type
and defining said electronic form so prepared as a computerized item;
4O preparing at least one session script having information indicative of predetermined test options and informa-
tion indicative of a sequence in which said computerized items are to be presented; and
packaging said computerized items and said session scripts together to form said computerized test.
26. The method of claim 25, wherein computerizing each item comprises the steps of_.
45 editing the items according to the presentation information using a word processor; and
inserting custom codes into the items according to both the presentation information and the response type.
27. The method of claim 25, wherein the items further comprise stimulus material to be references by the items, the
5O method fu_her comprising the step of_.
demarcating blocks of the stimulus material by inserting custom codes into the stimulus material;
said insertion of the custom codes causing the demarcated blocks to be one of i) highlighted upon presentation
to the examinees, and ii) selectable by the examinees.
55 28. The method of claim 25, wherein each computerized test has at least one test section, each test section having
at least one item, further comprising the steps of_. 1 47
EP O 664 O41 B1
preparing for each said section, at least one delivery unit defining options to be effective for that said section
and for specifying an organization of said computerized items to be presented within that said section.
29. The method of claim 28, further comprising the steps of_.
5 preparing at least one of the following units_.
(1 ) a general information screen (GIS) unit having information related to any one of said test, said section,
and said item, and
1O (2) a tutorial unit having information to familiarize examinees with said computerized test;
combining GIS units, tutorial units and delivey units to form a testing unit; and
generating a test script defining a sequence in which said GIS units, tutorial units and delivery units are to be
presented, at least one test script being included within each of said session scripts.
15 3O. The method of claim 29, further comprising the steps of_.
preparing at least one of the following units_.
2O (1 ) an examinee data collection unit having at least one inquiry to be presented to examinees to obtain
information about said examinees, and
(2) a scoring and reporting unit having information related to scoring and reporting of at least one com-
puterized test; and
25 defining a sequence in which said examinee data collection units, scoring and repo_ing units testing units,
GIS units, and tutorial units are to be presented, said session script being based on said sequence.
31 . The method of claim 25, further comprising the step of_.
3O generating computerized graphics to be presented with the computerized items.
32. The method of claim 25, wherein each item has a key indicative of a correct response tothe test question associated
therewith, the method further comprising the step of_.
35 computerizing the key; and
packaging the computerized key with the computerized test.
33. The method of claim 25, further comprising the step of_.
4O presenting each computerized item for review; and
modifying the computerized item based on said presentation.
34. The system of claim 1 , wherein the computerized test is for standardized testing and the computerized test has a
number of questions to be answered by an examinee and is based on a test specification which specifies any of
45 the following information_. the number of questions to be presented in the computerized test, the number of test
sections in which the questions are grouped, the number of questions in each section, an amount of time for taking
the test, an amount of time allotted for responding to each of the questions, and criteria for selecting questions to
be presented, said test specification being defined independent of said test development system.
5O 35. The method of claim 1 2, wherein the computerized test is for standardized testing and is produced using a test
development system, said computerized test having a plurality of questions to be answered by an examinee and
being based on a test specification which specifies any of the following information_. the number of questions to be
presented in the computerized test, the number of test sections in which the questions are grouped, the number
of questions in each section, an amount of time for taking the test, an amount of time allotted for responding to
55 each of the questions, and criteria for selecting questions to be presented, said test specification being defined
independent of said test development system;
delivering said computerized test to the examinee by presenting said plurality of questions to the examinee
148
EP O 664 O41 B1
at a computer workstation in a manner independent of the information specified by the test specification; and
recording responses provided by the examinee to the questions presented.
36. The apparatus of claim 1 , wherein the rules for determining the sequence of questions to be presented are based
5 on a measurement model for any one of a lineartest, an adaptive test, a randomized parallel test, and a masterytest.
37. The method of claim 1 2, wherein the rules for determining the sequence of questions to be presented are based
on a measurement model for any one of a lineartest, an adaptive test, a randomized parallel test, and a masterytest.
1O Patentansprü_he
1 . Testsystem auf Computerbasis, mit_.
15 einem Testentwicklungssystem (1 O) für das Erzeugen eines mit Computerhilfe erstellten Testes, welcher eine
Anzahl von Fragen hat, die von einem Prüfling zu beantworten sind, und welcher Regeln hat, um eine Folge
zu präsentierenden Fragen festzulegen,
einer Workstation (3), die so betreibbar ist, daß sie die Fragen zumindest einem Prüfling präsentiert, und
weiterhin so betreibbar ist, daß sie die auf die so dargestellten Fragen gegebenen Antworten des Prüflings
2O annimmt, und
einem Testabgabesystem (1 2), welches sowohl mit dem Testentwicklungssystem als auch mit der Workstation
wirksam verbunden ist, um den mit Computerhilfe erstellten Test abzugeben, indem die Fragen entsprechend
den Regeln jedem Prüfling an der Workstation präsentiert werden, dadur_h gekennzei_hnet, daß
der mit Computerhilfe erzeugte Test eine Mehrzahl von zueinander in Beziehung stehenden Testbildschirmen
25 hat, die _achrichten und Anweisungen enthalten, welche lnformationen für den Prüfling bereitstellen, und
gekennzeichnet durch ein mit Computerhilfe erstelltes Testscript (72O), welches unter Verwendung des Test-
entwicklungssystems (1 O) erzeugt wird, um die Workstation (3) zu steuern, damit sie die Fragen und die dazu
in Beziehung stehenden Testbildschirme entsprechend den durch das Testscript (72O) definierten Regeln be-
reitstellt.
3O 2. System nach Anspruch 1 , welches weiterhin aufweist_.
ein Testverwaltungssystem (1 4), welches mit dem Testentwicklungssystem wirksam verbunden ist, wobei das
Testabgabesystem und die Workstation für das Starten und Beenden der Abgabe des mit Computerhilfe er-
35 stellten Testes durch das Testabgabesystem vorgesehen sind.
3. System nach Anspruch 2, wobei das Testverwaltungssystem weiterhin eine Sicherheitseinrichtung für das Sichern
des unter Computerhilfe erstellten Testes gegenüber dem Zugriff durch nicht autorisierte Personen aufweist.
4O 4. System nach Anspruch 1 , wobei das Testentwicklungssystem weiterhin aufweist_.
eine Einrichtung (62) zur Erzeugung eines Testdokumentes, um die einem Prüfling zu präsentierenden Fragen
auf der Basis einer Testspezifikation (686) zu definieren, und um eine Art der Darbietung der Fragen auf der
Workstation während der Abgabe des mit Computerhilfe erzeugten Testes zu bestimmen,
45 einem Gegenstandsvorbereitungssystem (6O), welches über eine Schnittstelle mit der Einrichtung zur Erzeu-
gung des Testdokumentes verbunden ist, um die Fragen für die Abgabe durch das Testabgabesystem mit
Computerhilfe zu erstellen,
einem Testvorbereitungssystem (57), welches über eine Schnittstelle mit der Einrichtung zur Erzeugung eines
Testdokumentes und dem Gegenstandsvorbereitungssystem verbunden ist, um das mit Computerhilfe erstell-
5O te Testscriptum und die zugehörigen Testbildschirme vorzubereiten bzw. bereitzustellen, und
einer Hilfseinrichtun (58) zur Testzusammenstellung, die mit den Gegenstands- und Testvorbereitungssyste-
men über eine Schnittstelle verbunden ist, um die mit Computerhilfe erstellten Fragen, das Testscriptum und
die zugehörigen Testbildschirme miteinander zu kombinieren, um den mit Computerhilfe erstellten Test zu
bilden.
55 5. System nach Anspruch 4, wobei die Testbeschreibung eine Anzahl von Testabschnitten des mit Computerhilfe
erstellten (computerisierten) Tests angibt, in welchem die Fragen bezeichnet sind, wobei jedes Testscript die Re-
geln zum Festlegen der Folge von Fragen und der darzubietenden, zugehörigen Testbildschirme in einem der
1 49
EP O 664 O41 B1
Testabschnitte definiert, und wobei das Testvorbereitungssystem in der Weise wirksam ist, daß es zumindest eines
der folgenden erzeugt, und das Testabgabesystem so wirksam ist, daß es eines der folgenden abgibt bzw. liefert_.
Tutorien (71 4) für die Bereitstellung von Anweisungen an den Prüfling für die Interaktion mit der Workstation
5 während der Abgabe des mit Computerhilfe erstellten Tests,
Anweisungen (71 O) zur Erklärung, wie dieser computerisierte Test zu behandeln ist,
eine Hilfsfunktion (? HELP), die von dem Prüfling anwählbar ist, um auf zusätzliche Anweisungen zuzugreifen,
welche beschreiben, wie man mit der Workstation interagiert, und für den Zugriff auf die Anweisungen zur
Erklärung, wie der computerisierte Test anzuwenden ist,
1O eine Funktion zur nochmaligen Durchsicht (REVl EW), die für den Prüfling während der Abgabe des Compu-
tertests möglich ist, um eine Liste im wesentlichen all der Fragen des Computertests bereitzustellen, und um
es dem Prüfling zu ermöglichen, bestimmte Fragen aus der darzustellenden Liste auszuwählen und wobei
das Testvorbereitungssystem ein Sitzungsscriptum (71 8) erzeugt, welches eine Folge definiert, in welcher die
Tutorien, die Anweisungen und die Testscripten von dem Testabgabesystem aufgerufen werden sollen, und
15 wobei das Testabgabesystem so betreibbar ist, daß es die durch das Sitzungsscript definierten Sequenzen
bzw. Abfolgen und die auf der Basis der durch die Testscripten definierten Regeln bestimmten Abfolgen un-
terbricht, um bei Auswahl durch den Prüfling die Hilfefunktion und die Funktion zur nochmaligen Durchsicht
aufzurufen.
2O 6. System nach Anspruch 1 , wobei eine Mehrzahl von vorbestimmten Vorfällen durch das Testabgabesystem über-
wacht werden, wobei jeder Vorfall und zugehörige Information in einer Protokollaufzeichnung gespeichert werden,
wobei eine Kombination von Protokollaufzeichnungen, die während der Abgabe des computerisierten Tests er-
zeugt werden, eine Datei der Funktionsfähigkeit definieren.
25 _. System nach Anspruch 6, wobei die Vorfälle die Antworten des Prüflings beinhalten und das System weiterhin
aufweist_.
ein Datenverteilungssystem (1 8) mit einer Schnittstelle zu dem Testabgabesystem, um die Dateien der Funk-
tionsfähigkeit zu formatieren, so daß die Antworten des Prüflings ausgewertet werden können.
3O 8. System nach Anspruch 2, wobei eine Mehrzahl von Vorfällen, die Fehler des Testsystems auf Computerbasis
anzeigen und die sich auf die Sicherheit des computerisierten Tests und des Testsystems auf Computerbasis
beziehen, durch das Testverwaltungssystem überwacht werden, und wobei eine Mehrzahl von vorbestimmten
Vorfällen, die während der Abgabe des computerisierten Tests durch das Testabgabesystem auftreten, überwacht
35 werden, wobei jeder Vorfall und entsprechende Information innerhalb einer Protokollaufzeichnung gespeichert
werden, eine Kombination der Protokollaufzeichnungen, welche Vorfälle speichern, die sich auf Fehler beziehen,
eine Fehlerprotokolldatei definiert, eine Kombination der Protokollaufzeichnungen, welche Vorfälle speichern, die
sich auf die Sicherheit des computerisierten Tests und des Testsystems auf Computerbasis beziehen, eine Sicher-
heitsprotokolldatei definiert, und eine Kombination der Protokollaufzeichnungen, die Vorfälle speichert, welche
4O während der Abgabe des computerisierten Tests auftreten, eine Leistungsdatei des Prüflings definiert.
9. System nach Anspruch 8, wobei die Workstations an zumindest einem Testzentrum (2) angeordnet sind, und wobei
das System weiterhin aufweist_.
45 ein Datenverteilungssystem mit einer Schnittstelle mit dem Testverwaltungssystem, um die Fehlerprotokoll-
dateien, die Sicherheitsprotokolldateien und die Leistungsdateien des Prüflings zu verarbeiten, und um aus
diesen Dateien Berichte zu erzeugen, die die an jeder Workstation und an jedem der Testzentren auftretenden
Vorfälle anzeigen.
5O 1 O. System nach Anspruch 1 , wobei der computerisierte Test ein standardisierter Test ist.
1 1 . System nach Anspruch 1 , wobei die Workstations an zumindest einem Testzentrum angeordnet sind und das
System weiterhin aufweist_.
55 ein Datenverteilungssystem, welches an einer Zentralverarbeitungsstelle (1 ) angeordnet ist, die entfernt von
den Testzentren liegt, wobei das Datenverteilungssystem eine Datenverbindung mit jedem der Testzentren
hat, um Informationen, die sich auf Aktivitäten an den Testzentren beziehen, sowie Informationen, die die
Antworten des Prüflings anzeigen, aufzunehmen, und um die Informationen zu verarbeiten, um zumindest
1 5O
EP O 664 O41 B1
einen Bericht zu erzeugen, welcher diese Information in einer für Menschen lesbaren Form darbietet, und um
zumindest einige der Informationen für eine nachfolgende Bewertung bzw. Punktevergabe für die Antworten
des Prüflings zu formatieren.
5 1 2. Verfahren zum Testen auf Computerbasis, mit den Schritten_.
Erzeugen eines computerisierten Tests, der eine Mehrzahl von Fragen hat, die durch einen Prüfling zu beant-
worten sind, sowie Regeln zum Festlegen einer Abfolge von darzubietenden Fragen hat,
Abgeben des computerisierten Tests an den Prüfling,
1o Aufzeichnen von Antworten, die von dem Prüfling zu den dargebotenen Fragen gegeben werden,
dadur_h gekennzei_hnet, da_
der computerisierte Test eine Mehrzahl von dazugehörigen Testbildschirmen hat, die Nachrichten und Anwei-
15 sungen enthalten, welche für den Prüfling Informationen bereitstellen, und gekennzeichnet durch
ein computerisiertes Testscriptum (72O), welches unter Verwendung eines Testentwicklungssystems (1 O) er-
zeugt wird, das Regeln definiert für das Bestimmen einer Sequenz der Fragen und zugehöriger Testbildschir-
me, die dargestellt werden sollen, und dadurch, daß die Testabgabe erreicht wird durch Darbieten einer Mehr-
zahl von Fragen und zugehöriger Testbildschirme für den Prüfling an einer Computer-Workstation (3) entspre-
2O chend den durch das Testscript (72O) definierten Regeln.
1 3. Verfahren nach Anspruch 1 2, welches weiterhin den Schritt aufweist_.
Vorlegen zumindest eines der computerisierten Tests an einen Prüfling.
25 1 4. Verfahren nach Anspruch 1 2, welches weiterhin den Schritt aufweist_.
Sichern des computerisierten Tests und der Antworten gegenüber Zugriff durch eine nicht autorisierte Person.
3o 1 5. Verfahren nach Anspruch 1 2, wobei der computerisierte Test standardisiert ist und wobei der computerisierte Test
auf einer Testspezifikation (686) beruht, die unabhängig von dem Testentwicklungssystem definiert ist, welches
zur Herstellung des computerisierten Tests verwendet wird.
1 6. Verfahren nach Anspruch 1 5, wobei die Testspezifikation eine Anzahl von Testabschnitten des computerisierten
35 Tests näher beschreibt, welchen die Fragen zugeordnet werden, wobei jedes Testscript Regeln für das Bestimmen
der Folge von Fragen und zugehöriger Testbildschirme definiert, die in einem der Testabschnitte dargeboten wer-
den sollen, wobei der Schritt des Abgebens des computerisierten Tests weiterhin zumindest einen der folgenden
Schritte aufweist_.
4o Anzeigen von Tutorien bzw. Lernanweisungen für den Prüfling, um ihn darüber zu instruieren, wie er mit der
Workstation arbeiten (interagieren) soll,
Anzeigen von Anweisungen, die sich auf Fragen beziehen, um zu erläutern, wie der Prüfling auf jede der
dargebotenen Fragen antworten soll,
Bereitstellen einer Hilfsmöglichkeit, die durch den Prüfling auswählbar ist, damit er auf Anweisungen zugreifen
45 kann, die beschreiben, wie er mit der Workstation arbeiten soll, und das Zugreifen auf Anweisungen, die
beschreiben, wie er auf die dargebotenen Fragen antworten soll, und
Bereitstellen einer Möglichkeit zur erneuten Betrachtung, die durch den Prüfling auswählbar ist, um die dar-
gebotenen Fragen nochmals durchzusehen, indem eine Liste von im wesentlichen allen Fragen bereitgestellt
wird, die in dem computerisierten Test enthalten sind, und um dem Prüfling zu ermöglichen, zu einer der
5O aufgelisteten Fragen zurückzukehren bzw. weiterzugehen, und
wobei das Verfahren weiterhin die Schritte aufweist_.
Erzeugen eines Sitzungsscripts, welches eine Folge festlegt, in welcher die Tutorien, die Anweisungen und
die Testscripten aufgerufen werden sollen, und Unterbrechen der durch das Sitzungsscript festgelegten Ab-
folge und der Abfolge, die auf den durch die Testscripts definierten Regeln beruht, um die Hilfe- und Nach-
55 schaumöglichkeiten bei entsprechender Auswahl durch den Prüfling aufzurufen.
1 7. Verfahren nach Anspruch 1 2, wobei der Schritt des Abgebens des computerisierten Tests weiterhin die Schritte
aufweist_. 1 51
EP O 664 O41 B1
Bereitstellen von Benutzersteuerungen, die durch den Prüfer auswählbar sind für die Steuerung der Abgabe
des computerisierten Tests.
1 8. Verfahren nach Anspruch 1 2, wobei die Workstation einen Speicher aufweist und das Verfahren weiterhin die
5 Schritte aufweist_.
Registrieren zumindest einiger Prüflinge, um vor der Abgabe des computerisierten Tests einen spezifizierten
computerisierten Test aufzunehmen, wobei jeder auf diese Weise registrierte Prüfling autorisiert ist, den spe-
zifizierten Computertest durchzuführen,
1O Speichern zumindest eines spezifizierten computerisierten Tests in dem Speicher vor der Abgabe des spezi-
fizierten computerisierten Tests.
1 9. Verfahren nach Anspruch 1 8, welches weiterhin den Schritt aufweist_.
15 Sichern bzw. überwachen jedes computerisierten Tests, welcher in dem Speicher gespeichert ist, gegenüber
einem Zugriff durch nichtautorisierte Personen.
2O. Verfahren nach Anspruch 1 8, welches weiterhin den Schritt aufweist_.
2O Bestätigen bzw. Sicherstellen der Autorisierung für jeden Prüfling, der versucht, den spezifizie_en, compute-
risierten Test durchzuführen, bevor der spezifizierte, computerisierte Test abgegeben wird.
21 . Verfahren nach Anspruch 1 8, welches weiterhin die Schritte aufweist_.
25 Auswählen eines abzugebenden und in dem Speicher gespeiche_en, spezifizierten, computerisierten Tests,
wobei der ausgewählte Test derjenige spezifizierte, computerisierte Test ist, für welchen der Prüfling die Be-
rechtigung zur Entgegennahme hat, und
Beginnen mit der Abgabe des auf diese Weise ausgewähten, computerisierten Tests.
3O 22. Verfahren nach Anspruch 1 2, wobei jede derartige Workstation an einem Testzentrum angeordnet ist.
23. Verfahren nach Anspruch 22, welches weiterhin die Schritte aufweist_.
übermitteln von Informationen, die Aktivitäten anzeigt, welche an jedem der erwähnten Testzentren auftritt,
35 an eine zentrale Verarbeitungsstelle, die von den Testzentren entfernt liegt, und
Verarbeiten der übermittelten Informationen, so daß die Aktivitäten an den Testzentren an der zentralen Ver-
arbeitungsstelle überwacht werden.
24. Verfahren nach Anspruch 22, welches weiterhin die Schritte aufweist_.
4O übermitteln der aufgezeichneten Antworten von jeder Workstation an eine zentrale Verarbeitungseinheit, die
von den Testzentren entfernt liegt, und
Formatieren der aufgezeichneten Antworten für ein nachfolgendes Bewerten.
45 25. Verfahren nach Anspruch 1 2, wobei der Schritt des Herstellens eines computerisierten Tests die Schritte aufweist_.
Herstellen eines Testdokumentes mit Sachgebieten (Gegenständen), wobei jedes Sachgebiet zumindest eine
Testfrage hat, zugehörige Präsentationsinformation hat, die zumindest den einem Prüfling darzubietenden
Text und einen zu dem Sachgebiet gehörenden Antworttyp hat,
5O Herstellen eines elektronischen Formblattes für jedes Sachgebiet entsprechend der Darbietungsinformation
und dem Antworttyp und Definieren des so hergestellten elektronischen Formblattes als computerisierter Ge-
genstand,
Herstellen zumindest eines Sitzungsscriptums mit Information, die vorbestimmte Testinformationen anzeigt
und Information, die eine Folge anzeigt, in welcher die computerisierten Gegenstände bzw. Sachgebiete dar-
55 geboten werden sollen, und
Verpacken bzw. Zusammenfassen der computerisierten Gegenstände bzw. Sachgebiete und der Sitzungss-
cripte gemeinsam miteinander, um den computerisierten Test zu bilden.
1 52
EP O 664 O41 B1
26. Verfahren nach Anspruch 25, wobei das Computerisieren jedes Gegenstandes bzw. Sachgebietes die Schritte
aufweist_.
Editieren der Sachgebiete entsprechend der Darbietungsinformation unter Verwendung einer Textverarbei-
5 tung, und
Einfügen von kundenspezifischen Codes in die Sachgebiete entsprechend sowohl der Darbietungsinformation
als auch dem Antworttyp.
27. Verfahren nach Anspruch 25, wobei die Sachgebiete weiterhin Anregungsmaterial aufweisen, welche Bezugnah-
1O men auf die Gegenstände bzw. Sachgebiete sein sollen, wobei das Verfahren weiterhin den Schritt aufweist_.
Kennzeichnen von Blöcken des Anregungsmaterials durch Einfügen von kundenspezifischen Codes in das
Anregungsmaterial,
wobei die Einfügung der kundenspezifischen Codes bewirkt, daß die gekennzeichneten Blöcke entweder
15 i) bei der Darbietung an die Prüflinge hervorgehoben, und
ii) durch die Prüflinge aufwählbar sind.
28. Verfahren nach Anspruch 25, wobei jeder computerisierte Test zumindest einen Testabschnitt hat, jeder Testab-
2O schnitt zumindest ein Sachgebiet hat, und weiterhin die Schritte aufweist_.
für jeden Abschnitt Herstellen zumindest einer Abgabeeinheit, die Wahlmöglichkeiten definiert, die für diesen
Abschnitt wirksam sind, und zum Spezifizieren einer Organisation für die innerhalb des Abschnittes darzubie-
tenden, computerisierten Sachgebiete.
25 29. Verfahren nach Anspruch 28, welches weiterhin die Schritte aufweist_.
Herstellen zumindest einer der folgenden Einheiten_.
3O (1 ) einer allgemeinen Informationsbildschirmeinheit (GIS), die Information hat, die sich auf den Test, den
Abschnitt oder das Sachgebiet bezieht, und
(2) einer Lehreinheit, die Information hat, um die Prüflinge mit dem computerisierten Test vertraut zu ma-
chen,
35 Kombineren von Gl S-Einheiten, Lehreinheiten und Abgabeeinheiten, um eine Testeinheit zu bilden, und
Erzeugen eines Testscriptums, welches eine Folge definiert, in der die Gl S-Einheiten, die Lehreinheiten und
die Abgabeeinheiten dargeboten werden sollen, wobei zumindest ein Testscriptum in jedem der Sitzungss-
cripten eingeschlossen ist.
4O 3O. Verfahren nach Anspruch 29, welches weiterhin die Schritte aufweist_.
Herstellen zumindest einer der folgenden Einheiten_.
(1 ) einer Datensammlungseinheit des Prüflings, die zumindest eine den Prüflingen darzubietende Frage
45 enthält, um lnformation über die Prüflinge zu erhalten, und
(2) eine Bewertungs- und Berichteinheit, die Information hat, welche sich auf das Bewerten und das Be-
richten über zumindest einen computerisierten Test bezieht, und
Definieren einer Folge, in welcher dem Prüfling Datensammeleinheiten, Bewertungs- und Berichtseinheiten,
5O Testeinheiten, Gl S-Einheiten und Lehreinheiten dargeboten werden sollen, wobei das Sitzungsscript auf die-
ser Folge beruht.
31 . Verfahren nach Anspruch 25, welches weiterhin den Schritt aufweist_.
55 Erzeugen von Computergrafik, die mit den computerisierten Gegenständen dargeboten werden soll.
32. Verfahren nach Anspruch 25, wobei jedes Sachgebiet ein Schlüsselwort hat, welches eine korrekte Antwort auf
die zugehörige Testfrage hat, wobei das Verfahren weiterhin den Schritt aufweist_.
1 53
EP O 664 O41 B1
Computerisieren des Schlüssels, und
Verpacken des computerisierten Schlüssels mit dem computerisierten Test.
33. Verfahren nach Anspruch 25, welches weiterhin den Schritt aufweist_.
5 Darbieten jedes computerisierten Gegenstandes bzw. Sachgebietes für eine Betrachtung bzw. überprüfung,
und
Modifizieren des auf dieser Darbietung beruhenden, computerisierten Gegenstandes.
1O 34. System nach Anspruch 1 , wobei der computerisierte Test für ein standardisiertes Testen vorgesehen ist und wobei
der computerisierte Test eine Anzahl von durch einen Prüfling zu beantwortenden Fragen hat und auf einer Test-
spezifikation beruht, die irgendeine der folgenden Informationen speichert_. die Anzahl der in dem computerisierten
Test darzubietenden Fragen, die Anzahl der Testabschnitte, in welche die Fragen gruppiert sind, die Anzahl von
Fragen in jedem Abschnitt, eine Zeit für die Durchführung des Tests, eine Zeit, die jeweils der Beantwortung jeder
15 der Fragen zugeordnet ist, und Kriterien für die Auswahl von zu präsentierenden Fragen, wobei die Testspezifi-
kation unabhängig von dem Testentwicklungssystem definiert wird.
35. Verfahren nach Anspruch 1 2, wobei der computerisierte Test für standardisiertes Testen vorgesehen und unter
Verwendung eines Testentwicklungssystems hergestellt ist, wobei der computerisierte Test eine Mehrzahl von
2O durch einen Prüfling zu beantwo_enden Fragen hat und auf einer Testspezifikation beruht, die irgendeine der
folgenden Informationen spezifiziert bzw. näher angibt_. die Anzahl der in dem computerisierten Test zu präsentie-
renden Fragen, die Anzahl von Testabschnitten, in welche die Fragen gruppiert sind, die Anzahl von Fragen in
jedem Abschnitt, eine Zeit für die Durchführung des Tests, eine Zeit, die jeweils einer Beantwortung jeder der
Fragen zugeordnet ist, und Kriterien für die Auswahl von zu präsentierenden Fragen, wobei die Testspezifikation
25 unabhängig von dem Testentwicklungssystem definie_ wird,
Abgeben des computerisierten Tests an den Prüfling durch Präsentation der Mehrzahl von Fragen für den
Prüfling an einer Computer-Workstation in einer Art und Weise, die unabhängig von der durch die Testspezi-
fikation näher beschriebenen Information ist, und
3O Aufzeichnen von Antworten, die von dem Prüfling auf die gestellten Fragen gegeben werden.
36. Vorrichtung nach Anspruch 1 , wobei die Regeln für das Bestimmen der Folge von zu präsentierenden Fragen auf
einem Meßmodell für einen linearen Test, einen adaptiven Test, einen Paralleltest mit Zufallsauswahl oder einen
Haupttest beruht.
35 37. Verfahren nach Anspruch 1 2, wobei die Regeln für das Bestimmen der Folge von zu präsentierenden Fragen auf
einem Meßmodell für einen linearen Test, einen adaptiven Test, einen Paralleltest mit Zufallsauswahl oder einen
Haupttest beruht.
4O Revendi_ations
1 . Système d'épreuve assisté par ordinateur comprenant _.
45 un système de développement d'épreuve (1 O) pour créer une épreuve informatisée ayant un certain nombre
de questions auxquelles un candidat doit répondre et des règles pour déterminer une séquence de questions
à présenter ;
un poste de travail (3) pouvant être mis en oeuvre pour présenter les questions à au moins un dit candidat et
pouvant être mis en oeuvre pour accepter des réponses du candidat aux questions ainsi présentées ; et
5O un système de remise d'épreuve (1 2) relié de manière opérationnelle à la fois audit système de développement
d'épreuve et audit poste de travail pour effectuer la remise de ladite épreuve informatisée en présentant les
questions selon les règles à chaque candidat sur ledit poste de travail ; caractérisé en ce que
Iadite épreuve informatisée possède une pluralité d'écrans d'épreuve liés contenant des messages et des
directives donnant des informations au candidat ; et caractérisé par
55 un scénario d'épreuve informatisé (72O) créé en utilisant ledit système de développement d'épreuve (1 O) pour
commander ledit poste de travail (3) pour présenter les questions et les écrans d'épreuve liés, selon des règles
définies par ledit scénario d'épreuve (72O). 1 54
EP O 664 O41 B1
2. Système selon la revendication 1 , comprenant de plus _.
un système d'administration d'épreuve (1 4) relié de manière opérationnelle audit système de développement
d'épreuve, audit système de remise d'épreuve et audit poste de travail pour débuter et terminer la remise de
5 ladite épreuve informatisée par ledit système de remise d'épreuve.
3. Système selon la revendication 2, dans lequel ledit système d'administration d'épreuve comprend de plus des
moyens de sécurité pour bloquer l'accès à ladite épreuve informatisée à des personnes non autorisées.
1O 4. Système selon la revendication 1 , dans lequel ledit système de développement d'épreuve comprend de plus _.
des moyens de création de document d'épreuve (62) pour définir les questions à présenter au candidat sur
Ia base d'une spécification d'épreuve (686) et pour déterminer une manière de présenter les questions sur
Iedit poste de travail pendant la remise de ladite épreuve informatisée ;
15 un système de préparation d'éléments (6O) interfacé avec lesdits moyens de création de document d'épreuve
pour informatiser les questions devant être remises par ledit système de remise d'épreuve ;
un système de préparation d'épreuve (57) interfacé avec lesdits moyens de création de document d'épreuve
et ledit système de préparation d'éléments pour préparer le scénario d'épreuve informatisé et les écrans
d'épreuve liés ; et
2O un programme utilitaire d'épreuve (58) interfacé avec lesdits systèmes de préparation d'éléments et d'épreuve
pour combiner les questions informatisées, le scénario d'épreuve et les écrans d'épreuve liés pour former
l'épreuve informatisée.
5. Système selon la revendication 4, dans lequel la spécification d'épreuve spécifie un certain nombre de sections
25 d'épreuve de l'épreuve informatisée dans lesquelles les questions sont attribuées, chaque scénario d'épreuve
définissant les règles pour déterminer la séquence des questions et les écrans d'épreuve liés à présenter dans
l'une desdites sections d'épreuve et dans lesquelles ledit système de préparation d'épreuve est opérationnel pour
créer, et ledit système de remise d'épreuve est opérationnel pour remettre, au moins l'un des éléments suivants _.
3O des didacticiels (71 4) pour donner des instructions au candidat pour l'interaction avec ledit poste de travail
pendant la remise de ladite épreuve informatisée ;
des directives (71 O) pour expliquer comment mener ladite épreuve informatisée ;
un utilitaire d'aide (? HELP) pouvant être sélectionné par le candidat pour accéder à des instructions supplé-
mentaires décrivant comment interagir avec ledit poste de travail et pour accéder auxdites directives pour
35 expliquer comment mener l'épreuve informatisée ;
un utilitaire de révision (REVl EW) pouvant être sélectionné par le candidat pendant la remise de ladite épreuve
informatisée pour donner une liste de sensiblement toutes les questions de ladite épreuve informatisée et
pour permettre au candidat de sélectionner des questions spécifiques dans la liste à présenter ; et
Iedit système de préparation d'épreuve créant un scénario de session (71 8) définissant une séquence dans
4O laquelle les didacticiels, les directives et les scénarios d'épreuve sont à invoquer par le système de remise
d'épreuve et dans lequel le système de remise d'épreuve peut être mis en oeuvre pour interrompre les sé-
quences définies par le scénario de session et les séquences déterminées sur la base des règles définies par
Ies scénarios d'épreuve pour invoquer les utilitaires d'aide et de révision lors de la sélection par le candidat.
45 6. Système selon la revendication 1 , dans lequel une pluralité d'événements prédéterminés sont contrôlés par ledit
système de remise d'épreuve, chaque événement et chaque information correspondants étant stockés dans un
enregistrement de journal, une combinaison d'enregistrements de journal créés pendant la remise de ladite épreu-
ve informatisée définissant un fichier de performances.
5O _. Système selon la revendication 6, dans lequel lesdits événements comprennent lesdites réponses du candidat,
Ie système comprenant de plus _.
un système de répartition de données (1 8) ayant une interface avec ledit système de remise d'épreuve pour
formater lesdits fichiers de performances de sorte que lesdites réponses de candidat peuvent être évaluées.
55 8. Système selon la revendication 2, dans lequel une pluralité d'événements indicatifs d'erreurs dudit système
d'épreuve assisté par ordinateur et relatifs à la sécurité de ladite épreuve informatisée et dudit système d'épreuve
assisté par ordinateur sont contrôlés par ledit système d'administration d'épreuve, et une pluralité d'événements
155
EP O 664 O41 B1
prédéterminés se produisant pendant la remise de ladite épreuve informatisée sont contrôlés par ledit système
de remise d'épreuve, chaque événement et chaque information liée étant stockés dans un enregistrement de
journal, une combinaison desdits enregistrements de journal stockant des événements relatifs à des erreurs dé-
finissant un fichier de journal d'erreurs, une combinaison desdits enregistrements de journal stockant des événe-
5 ments relatifs à la sécurité desdites épreuves informatisées et du système d'épreuve assisté par ordinateur défi-
nissant un fichier de journal de sécurité, et une combinaison desdits enregistrements de journal stockant des
événements se produisant pendant la remise de ladite épreuve informatisée définissant un fichier de performances
du candidat.
1O 9. Système selon la revendication 8, dans lequel les postes de travail sont situés au niveau d'au moins un centre
d'épreuve (2), le système comprenant de plus _.
un système de répartition de données ayant une interface avec ledit système d'administration d'épreuve pour
traiter lesdits fichiers de journal d'erreurs, lesdits fichiers de journal de sécurité et lesdits fichiers de perfor-
15 mances du candidat et pour produire à pa_ir de ces fichiers des rappo_s indicatifs d'événements se produisant
au niveau de chaque poste de travail et au niveau de chacun desdits centres d'épreuve.
1 O. Système selon la revendication 1 , dans lequel ladite épreuve informatisée est une épreuve standardisée.
2O 1 1 . Système selon la revendication 1 , dans lequel lesdits postes de travail sont situés au niveau d'au moins un centre
d'épreuve, le système comprenant de plus _.
un système de répartition de données situé au niveau d'un site central de traitement (1 ) distant desdits centres
d'épreuve, ledit système de répartition de données ayant une connexion de données avec chaque centre
25 d'épreuve pour recevoir des informations relatives aux activités au niveau des centres d'épreuve et des infor-
mations indicatives desdites réponses de candidat et pour traiter lesdites informations pour produire au moins
un rapport présentant lesdites informations sous une forme compréhensible par l'homme et pour formater au
moins certaines desdites informations pour l'évaluation ultérieure des réponses de candidat.
3O 1 2. Procédé d'épreuve assisté par ordinateur, comprenant les étapes suivantes _.
production d'une épreuve informatisée ayant une pluralité de questions auxquelles un candidat doit répondre
et des règles pour déterminer une séquence de questions à présenter ;
remise de ladite épreuve informatisée au candidat ;
35 enregistrement des réponses données par le candidat aux questions présentées ; caractérisé en ce que
Iadite épreuve informatisée comporte une pluralité d'écrans d'épreuve liés contenant des messages et des
directives donnant des informations au candidat ; et caractérisé par
un scénario d'épreuve informatisé (72O), créé en utilisant un système de développement d'épreuve (1 O) dé-
finissant des règles pour déterminer une séquence des questions et des écrans d'épreuve liés à présenter ;
4O et en ce que ladite remise d'épreuve est effectuée en présentant ladite pluralité de questions et d'écrans
d'épreuve liés au candidat au niveau d'un poste de travail informatique (3) selon les règles définies par ledit
scénario d'épreuve (72O).
1 3. Procédé selon la revendication 1 2, comprenant de plus l'étape suivante _.
45 distribution d'au moins une dite épreuve informatisée à au moins un candidat.
1 4. Procédé selon la revendication 1 2, comprenant de plus l'étape suivante _.
5O blocage de l'accès à ladite épreuve informatisée et auxdites réponses à une personne non autorisée.
1 5. Procédé selon la revendication 1 2, dans lequel ladite épreuve informatisée est standardisée et dans lequel ladite
épreuve informatisée est basée sur une spécification d'épreuve (686) définie indépendamment dudit système de
développement d'épreuve utilisé pour produire l'épreuve informatisée.
55 1 6. Procédé selon la revendication 1 5, dans lequel la spécification d'épreuve spécifie un certain nombre de sections
d'épreuve de l'épreuve informatisée dans lesquelles les questions sont attribuées, chaque scénario d'épreuve
définissant des règles pour déterminer la séquence des questions et les écrans d'épreuve correspondants à pré-
1 56
EP O 664 O41 B1
senter dans l'une desdites sections d'épreuve, l'étape de remise de l'épreuve informatisée comprenant en outre
au moins une des étapes suivantes _.
- affichage de didacticiels destinés au candidat pour apprendre au candidat comment interagir avec ledit poste
5 de travail _,
- affichage de directives relatives aux questions pour expliquer comment le candidat est censé répondre à
chaque question présentée ;
- mise à disposition d'un utilitaire d'aide pouvant être sélectionné par le candidat pour accéder aux instructions
décrivant comment interagir avec ledit poste de travail et pour accéder aux directives décrivant comment
1O répondre aux questions présentées ; et
- mise à disposition d'un utilitaire de révision pouvant être sélectionné par le candidat pour réviser les questions
présentées en donnant une liste de presque toutes les questions que comporte l'épreuve informatisée et pour
permettre au candidat d'avancer et revenir à une question de la liste ; et
15 le procédé comprenant en outre les étapes consistant à _.
- créer un scénario de session définissant une séquence dans laquelle les didacticiels, les directives et les
scénarios d'épreuve sont à invoquer et
- interrompre la séquence définie par le scénario de session et la séquence déterminée basée sur les régles
2O définies par les scénarios d'épreuve pour invoquer les utilitaires d'aide et de révision sur une sélection du
candidat.
1 7. Procédé selon la revendication 1 2, dans lequel ladite étape de remise de ladite épreuve informatisée comprend
de plus l'étape suivante _.
25 présence de commandes utilisateur pouvant être sélectionnées par le candidat pour maîtriser ladite remise
de ladite épreuve informatisée.
1 8. Procédé selon la revendication 1 2, dans lequel ledit poste de travail comprend une mémoire, le procédé compre-
3O nant de plus les étapes suivantes _.
inscription d'au moins certains candidats pour mener une épreuve informatisée spécifiée avant la remise de
l'épreuve informatisée, chaque candidat ainsi inscrit étant autorisé à mener ladite épreuve informatisée
spécifiée ;
35 stockage d'au moins une épreuve informatisée spécifiée dans ladite mémoire avant la remise de ladite épreuve
informatisée spécifiée.
1 9. Procédé selon la revendication 1 8, comprenant de plus l'étape suivante _.
4O protection de chaque épreuve informatisée stockée dans ladite mémoire contre tout accès à des personnes
non autorisées.
2O. Procédé selon la revendication 1 8, comprenant de plus l'étape suivante _.
45 confirmation de l'autorisation de chaque candidat cherchant à mener ladite épreuve informatisée spécifiée
avant la remise de ladite épreuve informatisée spécifiée.
21 . Procédé selon la revendication 1 8, comprenant de plus l'étape suivante _.
5O sélection d'une épreuve informatisée spécifiée pa_iculière stockée en mémoire devant être remise, l'épreuve
sélectionnée étant ladite épreuve informatisée spécifiée que le candidat est autorisé à mener ; et
commencement de la remise de ladite épreuve informatisée ainsi sélectionnée.
22. Procédé selon la revendication 1 2, dans lequel chaque dit poste de travail est situé à un centre d'épreuve.
55 23. Procédé selon la revendication 22, comprenant de plus, les étapes suivantes _.
transfert d'informations indicatives d'activités se produisant à chaque centre d'épreuve depuis lesdits centres
1 57
EP O 664 O41 B1
d'épreuve jusqu'à un site central de traitement distant desdits centres d'épreuve ; et
traitement desdites informations transférées de sorte que lesdites activités auxdits centres d'épreuve sont
contrôlées au niveau dudit site central de traitement.
5 24. Procédé selon la revendication 22, comprenant de plus les étapes suivantes _.
transfert desdites réponses enregistrées en provenance de chaque poste de travail vers un site central de
traitement distant desdits centres d'épreuve ; et
formatage desdites réponses enregistrées pour une évaluation ultérieure.
1O 25. Procédé selon la revendication 1 2, dans lequel l'étape de production d'une épreuve informatisée comprend les
étapes suivantes _.
préparation d'un document d'épreuve d'éléments, chaque élément ayant au moins une question d'épreuve,
15 des informations de présentation liées indicatives d'au moins le texte à présenter à un candidat et une réponse
type associée audit élément ;
préparation d'une forme électronique de chaque élément selon lesdites informations de présentation et ladite
réponse type et définition de ladite forme électronique ainsi préparée en tant qu'élément informatisé ;
préparation d'au moins un scénario de session ayant des informations indicatives d'options d'épreuve prédé-
2O terminées et des informations indicatives d'une séquence dans laquelle lesdits éléments informatisés sont à
présenter ; et
conditionnement desdits éléments informatisés et desdits scénarios de session pour former ensemble ladite
épreuve informatisée.
25 26. Procédé selon la revendication 25, dans lequel l'informatisation de chaque élément comprend les étapes
suivantes _.
édition des éléments selon les informations de présentation en utilisant un traitement de texte ; et
insertion de codes de personnalisation dans les éléments à la fois selon les informations de présentation et
3O la réponse type.
27. Procédé selon la revendication 25, dans lequel les éléments comprennent de plus une matière d'encouragement
à référencer par les éléments, le procédé comprenant de plus l'étape suivante _.
35 délimitation de blocs de la matière d'encouragement en insérant des codes de personnalisation dans la matière
d'encouragement ;
Iadite insertion des codes de personnalisation amenant les blocs délimités soit i) à être mis en évidence lors
de la présentation aux candidats, soit ii) a pouvoir être sélectionnés par les candidats.
4O 28. Procédé selon la revendication 25, dans lequel chaque épreuve informatisée comporte au moins une section
d'épreuve, chaque section d'épreuve ayant au moins un élément, comprenant de plus les étapes suivantes _.
préparation pour chaque dite section, d'au moins une unité de remise définissant des options devant entrer
en vigueur pour cette dite section et pour spécifier une organisation desdits éléments informatisés à présenter
45 à l'intérieur de ladite section.
29. Procédé selon la revendication 28, comprenant de plus les étapes suivantes _.
préparation d'au moins une des unités suivantes _.
5O (1 ) une unité d'écran d'informations globales (Gl S) ayant des informations se rapportant à l'un quelconque
de ladite épreuve, de ladite section, et dudit élément, et
(2) une unité de didacticiel ayant des informations pour familiariser les candidats avec ladite épreuve
informatisée _,
55 combinaison des unités GIS, des unités de didacticiel et des unités de remise pour former une unité d'épreuve ;
et
production d'un scénario d'épreuve définissant une séquence dans laquelle lesdites unités Gl S, lesdites unités
1 58
EP O 664 O41 B1
de didacticiel et lesdites unités de remise sont à présenter, au moins un scénario d'épreuve étant inclus à
l'intérieur de chacun desdits scénarios de session.
3O. Procédé selon la revendication 29, comprenant de plus les étapes suivantes _.
5 préparation d'au moins une des unités suivantes _.
(1 ) une unité de collecte de données de candidat ayant au moins une demande de renseignements à
présenter à des candidats pour obtenir des informations concernant lesdits candidats, et
1O (2) une unité d'évaluation et de rapport ayant des informations se rapportant à l'évaluation et au rapport
d'au moins une épreuve informatisée ; et
définition d'une séquence dans laquelle lesdites unités de collecte de données de candidat, lesdites unités
d'évaluation et de rapport, lesdites unités d'épreuve, lesdites unités GIS, et lesdites unités de didacticiel sont
15 à présenter, ledit scénario de session étant basé sur ladite séquence.
31 . Procédé selon la revendication 25, comprenant de plus l'étape suivante _.
production de graphiques informatisés à présenter avec les éléments informatisés.
2O 32. Procédé selon la revendication 25, dans lequel chaque élément possède une clé indicative d'une réponse correcte
à la question d'épreuve associée à ce dernier, le procédé comprenant de plus l'étape suivante _.
informatisation de la clé _, et
25 enregistrement de la clé informatisée avec l'épreuve informatisée.
33. Procédé selon la revendication 25, comprenant de plus l'étape suivante _.
présentation de chaque élément informatisé pour une révision ; et
3O modification de l'élément informatisé sur la base de ladite présentation.
34. Système selon la revendication 1 , dans lequel l'épreuve informatisée sert à une épreuve standardisée et l'épreuve
informatisée comporte un certain nombre de questions auxquelles un candidat doit répondre et est basée sur une
spécification d'épreuve qui spécifie l'une quelconque des informations suivantes _. Ie nombre de questions à pré-
35 senter dans l'épreuve informatisée, le nombre de sections d'épreuve dans lesquelles les questions sont regrou-
pées, le nombre de questions dans chaque section, une quantité de temps pour mener l'épreuve, une quantité de
temps attribué pour répondre à chacune des questions, et des critères pour sélectionner les questions à présenter,
Iadite spécification d'épreuve étant définie indépendamment dudit système de développement d'épreuve.
4O 35. Procédé selon la revendication 1 2, dans lequel l'épreuve informatisée est destinée à une épreuve standardisée
et est produite en utilisant un système de développement d'épreuve, ladite épreuve informatisée ayant une pluralité
de questions auxquelles un candidat doit répondre et étant basée sur une spécification d'épreuve qui spécifie l'une
quelconque des informations suivantes ; le nombre de questions à présenter dans l'épreuve informatisée, le nom-
bre de sections d'épreuve dans lesquelles les questions sont regroupées, le nombre de questions dans chaque
45 section, une quantité de temps pour mener l'épreuve, une quantité de temps attribué pour répondre à chacune
des questions, et des critères pour sélectionner les questions à présenter, ladite spécification d'épreuve étant
définie indépendamment dudit système de développement d'épreuve ;
remise de ladite épreuve informatisée au candidat en présentant ladite pluralité de questions au candidat à
5O un poste de travail informatique d'une manière indépendante des informations spécifiées par la spécification
d'épreuve ; et
enregistrement des réponses données par le candidat à la question présentée.
36. Appareil selon la revendication 1 , dans lequel les règles pour déterminer la séquence de questions à présenter
55 sont basées sur un modèle de mesure pour l'une quelconque d'une épreuve linéaire, d'une épreuve adaptative,
d'une épreuve parallèle aléatoire et d'une épreuve de maîtrise.
37. Procédé selon la revendication 1 2, dans lequel les règles pour déterminer la séquence de questions à présenter
159
EP O 664 O41 B1
sont basées sur un modèle de mesure pour l'une quelconque d'une épreuve linéaire, d'une épreuve adaptative,
d'une épreuve parallèle aléatoire et d'une épreuve de maîtrise.
5
1O
15
2O
25
3O
35
4O
45
5O
55 16O
(PICTURE)
EP O 664 O41 B1
161
(PICTURE)
EPO664O41B1
F_G. 2
(PICTURE)
EP O 664 O41 B1 _
_
_
163
(PICTURE)
EP O 664 O41 B1 _
_
_
164
h0
_qq
_o _o_
_h_
_o_
____
_h_
_g (PICTURE)
_
EPO664O41 B1
_ _ h
_ _ iii _
__ (PICTURE)
o_ _
_ _\
_
_
_ h
0 _
\ _
h
0
h o
h
hq _
q0 _
q
__ o
_ _
g _\_ ___
_
,,,_
EP O 664 O41 B1
_ _
(PICTURE)
_ _ \_
_
_
_i _
__
_
_
_ _
_i .
0
(PICTURE)_
_\ 166
(PICTURE)
EP O 664 O41 B1 _
_
_
167
_
EP O664 O41 B1
OU7PU7 OF 7D/DC
7D(PICTURE)
(PICTURE)
7D
7O 7PS
74
7O 7PS
F_G_ 8
168
(PICTURE)
__
EPO664O41 B1
____ p_o_uç_/oN FuNC_oN__ F_OW D/_. GR_M
(PICTURE)
_lO _6
.. ,.. . FIG_ _
169
(PICTURE)
(PICTURE)(PICTURE)(PICTURE)
(PICTURE)(PICTURE)(PICTURE)
(PICTURE)
(PICTURE)(PICTURE)
(PICTURE)
EP O 664 O41 B1 _
_
_
_
17O
(PICTURE)
EP O 664 O41 B1
171
_(PICTURE)
EP O 664 O41 B1
172
(PICTURE)
EP O 664 O41 B1
173
___
__ (PICTURE)
EP O 664 O41 B1
174
(PICTURE)
EP O 664 O41 B1
175
(PICTURE)
EP O 664 O41 B1
176
(PICTURE)
EP O 664 O41 B1
177
__________
___(PICTURE)
EP O 664 O41 B1
178
(PICTURE)
EP O 664 O41 B1
179
(PICTURE)
EP O 664 O41 B1 _
_
_
_
18O
(PICTURE)
EP O 664 O41 B1 _
_
_
_
181
(PICTURE)
EP O 664 O41 B1
_
_h
O
h
_h
_
0
__
_
_
__
_i
_
_
__
h\
_
0
_
0
__ 182
(PICTURE)
EP O 664 O41 B1
__
__
_
M
M
_\
_
O
h_
_
_
_
_
_
_
_
_
M
Lu_
_
O
_
_
_
__
_
_
0
M
O_
_
M
__
_
_
M
_
0
_\
_
_
_
_ 183
(PICTURE)
EPO664O41B1
__ '
_
__
_
_
_\
_
_
hh
_
_
_
_
_
__
h
_
__
_ ___
ò _
_ _
_ _h
_ o_
__ ij_
h
_
0
_
__
_
_-
__
_
_
_
_
_
_h
_
__
_ 184
_
EPO664O41 B1
SESSlON SCR/P7S _N_ TEST SCRIPTS
(PICTURE)
Sf5SlON SCRlPT
(PICTURE)
SfSSlON SCRIP7 FIG_ 24
185
EP O664 O41 B1
7ES7 SCR P7S
J(PICTURE)
_E_ 7A_LES l
3
6O5
6O7
p_çKAçED 7ES T .
_JLfS
____ 25
186
(PICTURE)
EP O 664 O41 B1
_
_ _
_
__
_
_
h
_ q
_h _
_
_
_ _
O h_
_
h
_\
\
O
_
_
0
_
(PICTURE)
EPO664O41B1
_
_
_
_
_
q
_
h
_ _ _
_h __ _
i _
q _
h\ __
_ _
0 0 _h
_
_ _
h
__ _
_
0
__
0
_ _
_ _
___
__ò_
_____.
G __
_
EP O 664 O41 B1
_
O (PICTURE)
_
_
\
0
h\
_h
_
_
_
_
_
_ _
_ _
_\ .
h __
_ _
__
O
__
0
_
_
_
_
_
_
__
h\
EP O 664 O41 B1
_
O
_ (PICTURE)
_ (PICTURE)
\
0
h\
_
h
\
_
C__
_
_
_
_ _
_\ _
_ _
h
_ h_
_
\
G
__
O
_
_
_
_
_
_
h
_
h_
(PICTURE)
EPO664O41 B1
_
0
_
_
__.
_VV_;_
\
_
\\ __
V,
h _
\ _
_\_ __
0
_
_<
O
'_
_
_
_
_
h 191
_o___q_q__qo_
___h_\
_q_
__
_(PICTURE)
___
q___
EPO664O41B1
_
\
_\
__
_
q
_
h
_
_h_ ,,,
_q
o_\
___
__q
\_ __
_o
\_\
____
EP O664 O41 B1
(PICTURE)
_
_
\
\_
q
_
_
_q
h
_
_h
(PICTURE)
_(PICTURE)M 193
(PICTURE)
EP O 664 O41 B1
FIG. 3_
_ _
EP O664 O41 B1
(PICTURE)
_DMIN/_ 7R_ TlVf _PPL/C_ TlON 5l/
_DMIN/_ TR_ 7lVE _PPL/C_ 7l_N 5/l
FIG_ 3_ _
195
EP O 664 O41 B1
0 _ _
_ _ _
(PICTURE)
_ _ _ _
_
___
_
196
(PICTURE)
(PICTURE) (PICTURE)
(PICTURE)
(PICTURE)7EMp__7E/ (PICTURE)7EMPL_7F2
FIG. 34A FIG. 34E
7FMP__TF3 7FMPL_7f4
FIG. 34B FIG. 3dF
7EMp__7E_ 7FMPL_7E6
FIG. 34C (PICTURE)
, FIG. 34G
TFMP__7f7
PIG. 34D
(PICTURE)
EP O 664 O41 B1 _
_
__
_
198
(PICTURE)
EP O 664 O41 B1
199
(PICTURE)
EP O 664 O41 B1 _
_
_
_
.. _
2OO
(PICTURE)
EP O 664 O41 B1 _
_
_
__
_
2O1
(PICTURE)
EP O 664 O41 B1 __
_
_
_
2O2
(PICTURE)
EP O 664 O41 B1 __
_
__
_
2O3
(PICTURE)
EP O 664 O41 B1 __
_
_
_
2O4
(PICTURE)
(PICTURE)
(PICTURE)(PICTURE)(PICTURE)(PICTURE)(PICTURE)(PICTURE)
(PICTURE)
(PICTURE)(PICTURE)
EP O 664 O41 B1 U
_
_
_U
_
2O5
(PICTURE)
EP O 664 O41 B1
2O6
(PICTURE)
EP O 664 O41 B1
2O7
(PICTURE)
EP O 664 O41 B1
2O8
(PICTURE)
EP O 664 O41 B1
2O9
(PICTURE)
EP O 664 O41 B1 __
_
_
_
21O
EP O 664 O41 B1 0
_
(PICTURE)
(PICTURE)_ ' _
_
__
h_
211
EP O 664 O41 B1
(PICTURE)
' _ O0
_
_
_
212
(PICTURE)
EP O 664 O41 B1
213
(PICTURE)
EP O 664 O41 B1
214
EPO664O41 B1
_(PICTURE)
_
__
_(PICTURE)
'
_
__ h (PICTURE)
_
__
_ (PICTURE)
__ _
_
_(PICTURE)
_
__ ,
(PICTURE) _
_ _h
_
__ 0(PICTURE)
_
__
__ (PICTURE)
__ _(PICTURE)
_
__
_ (PICTURE)
'
_
__ 215
(PICTURE)
EP O 664 O41 B1 __
_
___
_
_o_
___
_
__
__
___
q_i
____
__
_o
___
___
o__
_____gq_
__\_
___
___
__
o
___i
___
o\
__q
__q_
_\_
_q
_q__ _
__
____
EPO664O41 B1
(PICTURE)
_
_
_
\
q
_
__ __i_
__q__
___
__
217
(PICTURE)
EP O 664 O41 B1 - _
_
__
_
218
(PICTURE)
(PICTURE)
F_G_ 4 4A
F_G _ 4 4_
(PICTURE)
EP O 664 O41 B1 _
_
_
___
_h
(PICTURE)
EP O 664 O41 B1 _
_
_
_
221
(PICTURE)
EP O 664 O41 B1 _
_
__
_
(PICTURE)
EP O 664 O41 B1
_ _
_
__
_
223
(PICTURE)_
_ _
_
EPO664O41 B1
(PICTURE)
ND_RD HE_DfR
O3
FIG. 48
EP O664 O41 B1
S7_R 7 SFSS/O_ D_ 7_ F/E_DS / qo2.
(PICTURE)
4
4/6
418
42O
422
424
426
428
FIG_ 4_
E/vD /7E_ D_ 7_ F/F_D / 4O_
(PICTURE)
43O
432
434
436
438
44O
442
444
FIG_ 5O
225
(PICTURE)
EPO664O41 B1
_DMIN/S_R_ _l__ SYS7_M
FUNC_lON_L FL OW Dl_GR_M
FIG_ 51 _
226
__
__
____ _
_
EP O 664 O41 B1
. H_ RD D/SK F/_ FS _
(PICTURE)
326 (PICTURE)
32O
(PICTURE)3(PICTURE)
FIG_ 52
(PICTURE)
EP O 664 O41 B1 3l6
6
GUfS _
MODf
FIG_ 53
(PICTURE)
EPO664O41 B1
(PICTURE)
ND_RD Hf_DfR
OO
_ FIG. 54
EP O 664 O41 B1
(PICTURE)
(PICTURE) _JOOJ
23O
_(PICTURE)
EP O 664 O41 B1
231
(PICTURE)
EP O 664 O41 B1
232
_(PICTURE)
EP O664 O41 B1
FIG. 5 7 .
(PICTURE)
EP O 664 O41 B1
234
__(PICTURE)
EPO664O41B1
' F_G.5_
(PICTURE)
EP O 664 O41 B1
236
(PICTURE)
EP O 664 O41 B1 FIG_ 61
(PICTURE)
EP O 664 O41 B1
238
(PICTURE)
EP O 664 O41 B1
239
(PICTURE)FIG .,,, 64
(PICTURE)_ F_G_ 6_Av
_(PICTURE)(PICTURE)
(///
EP O 664 O41 B1 O
(PICTURE)
/l/_
_ DMl_
C__Cf_ S
C_ O(PICTURE)
Sf
(PICTURE) FIG_ 65B
242
(PICTURE)
EP O 664 O41 B1
FIG. 66
(PICTURE)
EP O 664 O41 B1
(PICTURE)''' .. FIG. 6 7
(PICTURE)
EP O 664 O41 B1 _
(PICTURE)
I/5 7
F_G. 68 _
__
__ _(PICTURE)
EP O 664 O41 B1
FIG_ 6_
(PICTURE)
EP O 664 O41 B1
Il63. P_G_ PO
(PICTURE)
(PICTURE)(PICTURE)
(PICTURE)FIG. P1
(PICTURE)_ FIG. P2
(PICTURE)
EP O 664 O41 B1
249
(PICTURE)
F_G_ P3_
(PICTURE)_
EP O 664 O41 B1
_ FIG. P4
_ _ (PICTURE)
EP O 664 O41 B1
FIG_ P5
(PICTURE)
EP O 664 O41 B1
FIG_ P6
(PICTURE)
EP O 664 O41 B1
FIG. 77
(PICTURE)
EP O 664 O41 B1
J_O 78
255
(PICTURE)
EP O 664 O41 B1
FIG. P_
_ (PICTURE)
EPO664O41B1
FIG. 8O
(PICTURE)
EP O 664 O41 B1
l_J5 (PICTURE)
F_G_ 81.
(PICTURE)
EP O 664 O41 B1
(PICTURE)
l24O
259
(PICTURE)
(PICTURE)(PICTURE)
j
EP O 664 O41 B1
J262 (PICTURE)
l27O _ FIG. 83B
(PICTURE)
EP O 664 O41 B1 84
'''' FIG_ 84
(PICTURE)(PICTURE)
/32 O
_ 8 5
(PICTURE)(PICTURE)(PICTURE)(PICTURE)
/32 5
_ FIG _ 8 6
(PICTURE)
EP O 664 O41 B1
263
(PICTURE)
_
EP O 664 O41 B1 O0
_
_
__
(PICTURE)
EP O 664 O41 B1
_
__
_
_
_
__
_
_
_
0
_
_
_
_
\_ _
_
_
,,, _
(PICTURE)
EP O 664 O41 B1
266
(PICTURE)(PICTURE) F_G_ _OB
(PICTURE)
EP O 664 O41 B1 4
46
FIG_ _OC
(PICTURE)
_
EP O 664 O41 B1 _
_
___
269
(PICTURE)
EP O 664 O41 B1 FI__ _2A
(PICTURE)
(PICTURE)
EP O 664 O41 B1 8
7O
(PICTURE) 2O 72
2O 74
2O 76
2O 78
F_G_ _2_
(PICTURE)
_
_
EP O 664 O41 B1 _
_
___
272
(PICTURE)
EP O 664 O41 B1
(PICTURE)
_O88
2O9O
FIG_ _4A
(PICTURE)
_
- FIG_ _dB
(PICTURE)
_
EP O 664 O41 B1 ' _
_
___
_
275
(PICTURE)
EP O 664 O41 B1 FIG_ _6
(PICTURE)
EP O 664 O41 B1 _
_
__
h_
277
(PICTURE)
EP O 664 O41 B1 _7_
_/84 (PICTURE)
_ FIG. _8
(PICTURE)
_
EP O 664 O41 B1 __
___
(PICTURE)
EPO664O41B1
(PICTURE) q _ .
(PICTURE)
0 0
0 0
_ ' _
__ __
h_ h_
__ ___ _
_ __ _
_ _
_ _
_ _ __
_ _ j
_ _ iu
_ _ _
_(PICTURE)
__ _
EPO664O41B1
(PICTURE)
(PICTURE) (PICTURE)
0 (PICTURE)
-
0
O
_
o'
O
___oO 0
__0o 0'
___o 0
_ ____ 0o
__ ___0o_ .
_ __0O ___
_ jj_0o \_
ij___ Oj
__0o ~q
_' 00 _
_ .. o
_ ___ _
\ oi _ g
_ _O \ 0
_ __ 0 ~
0M_~
_ __i_
O hqo
_ _ci_q_ \
_ \ __ __ j_ _
___ M ij
oo , ___ h _
____00 __ q_
o ___00 __ _
o ___00 __ _ _
0 ___0'0' _jj Oi_
o ___OO __ _
0 __oo __ ,
o ___o, _ j _
0o _jj_0o _.. ~_
o ___ o ___ _ _
0 ______0 _o _
______.. o __ __
__ ___ _\ x __ __
_ ____ o _
h .__j0 _ __ _
\ ____ _~ _ __q _
iu ,x__ _ _ h_
_ _____g __ __.
____iu _ __ \
M _ _qij M __ o_
M ___>ihO _ _ _
_q ____M_ ò \ _
___ _0O Oij \
____ q0 h _ 0 ..
_____ ___ q ___ _ _
__jj_ __ g ._ q _j
___0 O ij ___ g _
\O ____o _\_. O__ ij iu
O 0O ____o \__ _ __ jj j
h 0 _\___o g \ ___ _ _
q iu ____O, ___ _ __ \_ M
g _. ___0 _q _ __
_ ___ ____o g o _ \ \
ii _ _i_____ jo _ _ o Q
_\ _ jj____o _o q_ M MM
_ \ .. _o 0 h __ij M _
_ M \ _ _i g g
_ _ M __. o _ _ \ M M
_ ____ i M_ \_o i \
h \ __ i. _ __ _ __ q q
\ h _Cj h0 q_ _ o _ ii
iu _ q_ _~ __h q\ h h
_ _ iq __ q _
281
(PICTURE)
(PICTURE)(PICTURE)
____
EPO664O41B1
FN7FR PROÇ7OR INFORM_7/(PICTURE)
_DM LOGON /D._ XXXXXXXX (PICTURE)
N_Mf._ (FJ XXXXXXXXXX
_U7H LF_FL._ /
F_ - S__F F3 - DF_F7F
,.,..,_ F_G_ 1 OOD
_
FN7FR P_CK_GF fON7ROL /(PICTURE)
P_CK_GF CON7RO_ _FRSION (PICTURE)
F2 - S__f F3 - DfLf7f
F_G. 1OO_
F_G. 1 OOF