SimStudent Logging Schema

Logging Documentation for SimStudent V 6.X

Color Code Key:

ActionType

Action Name

Action Actual Text

Other Custom Field

Standard DataShop Field

Actual Text Appearing

I. Each field in a logging transaction is stored in a separate column in DataShop.

a. The most general column holds an ActionType, each of which has a set of actions assigned to it, stored in a different DataShop column. The possible ActionTypes are as follows:

i. SIM_STUDENT_DIALOGUE

ii. SIM_STUDENT_INFO_RECEIVED

iii. SIM_STUDENT_LEARNING

iv. SIM_STUDENT_ERROR

v. SIM_STUDENT_PLE

vi. SIM_STUDENT_QUIZ

vii. SIM_STUDENT_ACTION_LISTENER

viii. SIM_STUDENT_STEP

ix. SIM_STUDENT_PROBLEM

x. SIM_STUDENT_EXPLANATION

II. SIM_STUDENT_DIALOGUE is used previously with dialogue interactions with SimStudent and now only with interactions that do not log receipt of information.

a. HINT_REQUEST_ACTION (Sim Student Requesting Hint) is logged when a hint is requested but not yet received.

i. STEP

ii. KC Category(Default) holds the skill name of the expected skill the hint should contain

iii. KC(Default) holds the skill name from KC Category(Default) suffixed with –SimStudent

iv. Feedback holds SimStudent’s request

v. Outcome is HINT

vi. Tutor Response Type is HINT_MSG

vii. Student Response Type is HINT_REQUEST and Student Response Subtype is SimStudent

viii. USERID

III. SIM_STUDENT_INFO_RECEIVED is used when some data is input by the user.

a. HINT_RECEIVED (Hint Received) is logged when a hint has been received from the user.

i. STEP

ii. SAI is the SAI of the hint entered and logged in Selection, Action and Input

iii. ACTUAL_SKILL holds the name of the skill which the student applied

iv. DURATION is the time since the hint was requested

v. KC Category(Default) holds the skill name of the expected skill the hint should contain

vi. KC(Default) holds the skill name from KC Category(Default) suffixed with –Student

vii. Outcome contains CORRECT if the hint given by the student is a correct answer or INCORRECT if the hint is incorrect

viii. Student Response Type is ATTEMPT and Student Response Subtype is Student

ix. Tutor Response Type is RESULT and Tutor Response Subtype is Logged Only

x. USERID

b. CONFIRMATION_REQUEST_ACTION (Sim Student Requesting Confirmation of Input) is logged when SimStudent requests a response to a SimStudent performed step.

i. STEP

ii. Feedback is logged with the text of the query to the user

iii. SAI in Selection, Action and Input are the SAI suggested by SimStudent

iv. KC Category(Default) holds the skill name of the expected skill that should have been applied at that step

v. KC(Default) holds the skill name from KC Category(Default) suffixed with –SimStudent

vi. ACTUAL_SKILL holds the name of the skill which SimStudent did apply

vii. Outcome holds the correctness of the input which SimStudent is suggesting

viii. Student Response Type is ATTEMPT and Student Response Subtype is SimStudent

ix. Tutor Response Type is RESULT and Tutor Response Subtype is Logged Only

x. USERID

c. INPUT_VERIFY_ACTION (Student Verified Correctness of Sim Student Input) is logged when the response to a SimStudent performed step is received.

i. STEP

ii. SAI in Selection, Action and Input are the SAI suggested by SimStudent

iii. RESULT contains Correct Action if the student answered Yes to SimStudent’s query and Error Action if the student answered No

iv. KC Category(Default) holds the skill name of the expected skill that should have been applied at that step

v. KC(Default) holds the skill name from KC Category(Default) suffixed with –Student

vi. ACTUAL_SKILL holds the name of the skill which SimStudent did apply

vii. DURATION holds the time (ms) since the verification was requested

viii. Outcome is logged with CORRECT if SimSt provided a correct SAI and Student response was Correct Action or if SimSt provided an incorrect SAI and student response was Error Action. Outcome is logged with INCORRECT otherwise.

ix. Student Response Type is ATTEMPT and Student Response Subtype is Student Meta

x. Tutor Response Type is RESULT and Tutor Response Subtype is Logged Only

xi. USERID

d. BAD_INPUT_RECEIVED (Invalid Input Received) is used when the Input Checker has rejected some user input.

i. STEP

ii. RESULT is logged with what the input is changed to, if the Input Checker interprets it. Otherwise, this field is blank.

iii. SAI in Selection, Action and Input are the SAI of the invalid, rejected input.

iv. ACTUAL_SKILL is the skill name that the student input attempted to apply if it can be determined and UNKNOWN otherwise

v. Student Response Type is HINT_REQUEST and Student Response Subtype is Student

vi. Tutor Response Type is HINT_MSG, Feedback is the error messaging given, and Outcome is HINT only in the cases where the bad input is not automatically corrected

vii. USERID

IV. SIM_STUDENT_LEARNING is used to log statistics and steps in the learning process.

a. WME_PATH_SEARCH_PERFORMANCE_ACTION (WME Path Search Performance) logs when the WME path is searched when the rules are generated, such as after a hint is provided

i. RESULT is a description of the statistic: WME paths Search for skillName found numberOfPaths paths

ii. DURATION contains the time (ms) it took to complete the search

iii. USERID

b. RHS_OPS_SEARCH_PERFORMANCE_ACTION (RHS Ops Search Performance) logs when the right hand side is searched to generate a rule

i. RESULT holds a description of the statistic: RHS Ops Search for skillName found numberOfOperators operators

ii. DURATION contains the time (ms) it took to complete the search

iii. USERID

c. RULE_LEARN_PERFORMANCE_ACTION (Rule Learned Performance) logs when a rule is learned when the rules are generated, such as after a hint is provided

i. RESULT holds a description of the statistic: skillName has been learned numberOfTimes times

ii. DURATION contains the time (ms) it took to learn the rule

iii. USERID

d. ACTIVATION_RULE_ACTION (Activation Rules Determined) logs when the activation list of rules fired is created

i. STEP

ii. RESULT holds the number of rules found

iii. DURATION contains the time (ms) it took to gather the activation list

iv. USERID

e. NEGATIVE_EXAMPLE_ACTION (Negative Example Received) logs when an instruction is processed as incorrect after the student labels it as an Error Action

i. RESULT gives the FOA used in determining the instruction

ii. USERID

f. POSITIVE_EXAMPLE_ACTION (Positive Example Received) logs when an instruction is processed as correct after the student labels it as a Correct Action

i. RESULT gives the FOA used in determining the instruction

ii. USERID

g. NOT_LEARN_ACTION (SimStudent Did Not Learn on Input) logs when SimStudent fails to learn on some instruction.

i. STEP

ii. RESULT gives the number of retries remaining to give input for SimStudent to attempt to learn on

iii. SAI in Selection, Action, Input holds the SAI of the instruction on which learning failed

iv. ACTUAL_SKILL is the skill name that the student input attempted to apply

v. DURATION contains the time (ms) that was spent attempting to learn the rule

vi. Feedback is logged with the text of the error given to the user

vii. Student Response Type is HINT_REQUEST and Student Response Subtype is Student

viii. Tutor Response Type is HINT_MSG

ix. USERID

V. SIM_STUDENT_ERROR is used for logging exceptions which occur and are caught within the SimStudent code.

a. EXCEPTION_ACTION (Sim Student has experienced an exception) indicates than an exception occurred and was caught

i. RESULT contains an explanation of the exception from the code, if given, and the message attached to the exception. Explanation-message (or only message)

ii. USERID

VI. SIM_STUDENT_PLE is used with notifications about the PLE itself

a. PLE_STARTED_ACTION (Peer Learning Environment Started) logs when the PLE successfully starts up

i. USERID

VII. SIM_STUDENT_QUIZ logs information about SimStudent’s progress on a quiz

a. CONFIRMATION_REQUEST_CL_ACTION (ClAlgebra Tutor Verified Correctness of Sim Student Input) logs when SimStudent has performed a step on a quiz problem and received response from CLTutor as to its correctness

i. STEP

ii. RESULT logs CLTutor’s response as to the correctness of the instruction, being either Correct Action or Error Action

iii. SAI in Selection, Action, Input contains the SAI of the instruction which SimStudent attempted to use

iv. Outcome holds the correctness of the input which SimStudent attempted

v. Student Response Type is ATTEMPT and Student Response Subtype is SimStudent

vi. Tutor Response Type is RESULT and Tutor Response Subtype is Quiz

vii. KC Category(Default) holds the skill name of the expected skill that should have been applied at that step

viii. KC(Default) holds the skill name from KC Category(Default) suffixed with –SimStudent

ix. ACTUAL_SKILL holds the name of the skill which SimStudent did apply

x. USERID

b. QUIZ_QUESTION_GIVEN_ACTION (Sim Student Given Quiz Question) logs when SimStudent is given another quiz question, before its begins it

i. STEP indicates the quiz problem completed by Quiz#.#, where the first # is the quiz section the problem was located in, and the second # is the index of the problem out of all quiz questions, starting at 0

ii. USERID

c. QUIZ_QUESTION_ANSWER_ACTION (Sim Student Answers Quiz Question) logs after SimStudent has just completed the answer to a quiz question

i. STEP indicates the quiz problem completed by Quiz#.#, where the first # is the quiz section the problem was located in, and the second # is the index of the problem out of all quiz questions, starting at 0

ii. RESULT holds the steps which were taken in SimStudent’s solution. This includes only instructions which were actually taken, not all which were tried and rejected. If all instructions are rejected, only the last one attempted is displayed.

iii. SIMST_CORRECTNESS contains whether SimStudent’s answer to the quiz question was correct and complete or not, holding either TRUE or FALSE

iv. DURATION is the time (ms) taken to complete that question

v. Outcome holds the correctness of the entire problem

vi. Tutor Response Type is RESULT and Tutor Response Subtype is Logged Only

vii. USERID

d. QUIZ_COMPLETED_ACTION (Quiz Completed) indicates that all of the questions on the quiz have been answered and SimStudent is now done attempting the quiz

i. STEP indicates the quiz section completed by Quiz#, where # is the section number of the quiz taken

ii. RESULT holds the percent of quiz questions which SimStudent answered correctly and completely out of all the quiz sections as a decimal value. Previously completed sections are included as correct in the total.

iii. DURATION holds the time it took to complete all questions on the quiz

iv. USERID

VIII. SIM_STUDENT_ACTION_LISTENER logs when the user manipulates the UI elements of the PLE

a. NEXT_PROBLEM_BUTTON_ACTION (Next Problem Button Clicked) logs when the user clicks the button to move to a new problem

i. USERID

b. QUIZ_BUTTON_ACTION (Quiz Button Clicked) logs when the user clicks the button to quiz SimStudent

i. USERID

c. TAB_SWITCH_ACTION (Switched Example Tab) logs when the user switches to a different tab on the PLE, paired with a TAB_LEFT_ACTION

i. RESULT logs the title of the new selected tab

ii. USERID

d. TAB_LEFT_ACTION (Left Example Tab) logs when the user switches away from a tab on the PLE, paired with a TAB_SWITCH_ACTION

i. RESULT logs the title of the tab which is being left

ii. DURATION logs the time since the user switched to that tab

iii. USERID

e. RESTART_BUTTON_ACTION (Restart Problem Button Clicked) logs when the user clicks the button to restart the problem

i. USERID

f. UNDO_BUTTON_ACTION (Undo Button Clicked) logs when the user clicks the button to undo the previous step

i. USERID

IX. SIM_STUDENT_STEP is used to log steps which SimStudent encounters

a. STEP_STARTED_ACTION (New Step Started) logs when SimStudent begins work on a new step of a problem

i. STEP

ii. USERID

b. STEP_COMPLETED_ACTION (Step Completed) logs when SimStudent finishes a step of a problem and moves onto the next

i. STEP

ii. DURATION contains the time (ms) which was spent working on the step

iii. USERID

c. UNDO_ACTION (Step Undone) logs when a step is undone after prompting from the Erase Last Step button

i. STEP the step which was undone

ii. RESULT can be Was Done, Was FailToLearn Stop, Was Normal HintRequest, indicating whether the state that is being reverted from was a done state, a given-up state from no retries remaining after failures to learn, or otherwise, respectively.

iii. Outcome holds the correctness of removing the step, which is the opposite of the original correctness of the step

iv. Student Response Type is ATTEMPT and Student Response Subtype is Student Meta

v. Tutor Response Type is RESULT and Tutor Response Subtype is Logged Only

vi. KC Category(Default) holds the skill name of the expected skill that should have been applied at that step

vii. KC(Default) holds the skill name from KC Category(Default) suffixed with –Student

viii. USERID

X. SIM_STUDENT_PROBLEM logs problems which SimStudent encounters

a. PROBLEM_ABANDONED_ACTION (Problem Abandoned) logs when SimStudent begins work on a new problem without finishing the previous one

i. STEP

ii. DURATION is the time (ms) since the problem was started

iii. USERID

b. PROBLEM_ENTERED_ACTION (New Problem Entered) is used when a new problem is entered and the start state is created

i. STEP is the name of the problem (i.e. the first step of it)

ii. USERID

c. PROBLEM_RESTART_ACTION (Problem Restarted) is used when SimStudent starts working on the problem again after the restart button is clicked

i. STEP is the name of the problem (i.e. the first step of it)

ii. USERID

d. PROBLEM_COMPLETED_ACTION (Problem Completed) is used when a problem is finished (the done state is accepted)

i. STEP

ii. DURATION is the time (ms) since the problem was started

iii. USERID

XI. SIM_STUDENT_EXPLANATION is used when an explanation for input is provided by the user for self-explanation items.

a. HINT_EXPLAIN_ACTION (Hint Explained) is logged when an explanation for a hint has been received from the user.

i. STEP

ii. RESULT is logged with the explanation which is given

iii. SAI is the SAI of the hint entered and logged in Selection, Action and Input

iv. DURATION holds the time (ms) since the explanation was requested

v. ACTUAL_SKILL holds the skill used by the SAI being explained

vi. USERID

b. INPUT_WRONG_EXPLAIN_ACTION (Non-Confirmed Input Explained) is logged when an ErrorAction response to a SimStudent performed step is explained.

i. STEP

ii. RESULT is logged with the explanation which is given

iii. SAI in Selection, Action and Input are the SAI suggested by SimStudent

iv. DURATION holds the time (ms) since the explanation was requested

v. ACTUAL_SKILL holds the skill used by the SAI being explained

vi. USERID

c. NO_EXPLAIN_ACTION (Explanation Not Given) is used when SimStudent has requested an explanation from the user and the user has cancelled the dialog

i. STEP

ii. RESULT holds the type of explanation requested. Can be either ProblemEntered, InputWrong, HintGiven

iii. SAI in Selection, Action and Input are the SAI of the item requiring explanation for InputWrong or HintGiven. Not present for ProblemEntered.

iv. DURATION holds the time (ms) since the explanation was requested

v. ACTUAL_SKILL holds the skill used by the SAI requested to be explained for InputWrong or HintGiven. Not present for ProblemEntered.

vi. USERID

d. PROBLEM_ENTERED_EXPLAIN_ACTION (New Problem Explained) is logged when a new problem’s entry is explained

i. STEP

ii. RESULT is logged with the explanation which is given

iii. DURATION holds the time (ms) since the explanation was requested

iv. USERID

XII. Default Logging Properties – CTAT handles much of the structure of the log messages, including the following items. We do not set most of these values, but they should be present in every log message if the CTAT implementation does not change.

a. Student Id – anonymous UserID from our own CF(USERID) in local logs, the same re-anonymized in DataShop

b. Session Id – unique to instance SimStudent is started up

c. Time – Date & Time

d. Time Zone

e. Level(Unit) – set as command line argument –Dunit_name.

f. Problem Name – attached to context message, we send a new context every time we receive a log with a PROBLEM_ENTERED_ACTION action.

g. School – set as command line argument –Dschool_name

h. Class – set as command line argument –Dclass_name

i. Condition – contains either selfExplain Enabled or selfExplain Disabled depending on the state of the command line flag -ssSelfExplainMode, indicating whether the experimental condition for the Self Explanation study is on or off

XIII. Specific Use of standard DataShop columns – As found in DataShop logs

a. Student Response Type – ATTEMPT by default or HINT_REQUEST when expecting a response from the system to the logged event that is not evaluated, such as a primarily feedback message

b. Student Response Subtype – describes which agent completed the action. It can be either Student for a student-completed action, SimStudent for a SimStudent-completed action, or Student Meta for a student evaluation or manipulation of a SimStudent-completed action

c. Tutor Response Type – matched with Student Response Type, RESULT pairs with ATTEMPT, HINT_MSG pairs with HINT_REQUEST

d. Tutor Response Subtype – indicates the result which the tutor’s evaluation has on the execution process. It can be either Logged Only, which indicates that the evaluation is noted in the logs but does not affect execution, or it can be Quiz, which indicates that it is a step in the evaluation of a quiz problem and that action may be rejected based on the tutor response

e. Step Name – indicates the cell or button which should have been used based on expected SAI according to CLTutor. For example dorminTable3_C1R1 UpdateTable is in the Step Name field when it is time to enter the initial transformation. This contrasts with CF(STEP) which is the step of the problem rather than the tutoring execution

f. Attempt At Step – the number of times including the one in the current log message which the step in the Step Name column has been tried for both Student and SimStudent for that problem

g. Outcome – an evaluation of the correctness of the step. The agent which is correct is the one which is listed in Student Response Subtype. It can be HINT for unevaluated messages, CORRECT for correct actions, or INCORRECT for incorrect actions.

h. Selection, Action and Input hold the SAI of the action which was completed. Selection is the component it was done to, Action is what type of action was done, and Input is the value used.

i. Feedback - messaging which is given to the student from SimStudent, often a specific question or request.

j. KC(Default) – the skill name which is expected by CLTutor, further divided into whether it was the student or SimStudent that applied some skill and should have applied the given skill by a suffix of either –Student or –SimStudent

k. KC Category(Default) – the same skill name from KC(Default) without the subdivision for student or SimStudent.

XIV. Custom Fields – As found in DataShop logs

a. CF(ACTION) – In all SimStudent log entries. Secondary field in the hierarchical organization. Always describes the event that is being logged.

b. CF(ACTION_TYPE) – In all SimStudent log entries. Primary field in the hierarchical organization. Always describes the broad type of the event that is being logged for organizational purposes.

c. CF(ACTUAL_SKILL) – The skill which is actually applied to get the result or UNKNOWN if unable to be determined when present.

d. CF(DURATION) – The time in milliseconds associated with the action described in the log message.

e. CF(RESULT) – Multipurpose field. Use varies based on action. Can be any sort of information resulting from the logged event.

f. CF(STEP) – Always the current state of SimStudent when the logged action occurred, when present. Represented by the last full state of the problem, and any transformation which has been applied to that problem state, not to get to that state.

g. CF(USERID) – In all SimStudent log entries. Always the user ID of the person using the system at the time the log action occurs, as given to SimStudent when it launches.

h. CF(tool_event_time) – Not used by SimStudent log entries. Created by CTAT for its log entries which do not go through SimStudent logging.