June 16, 2010

Accountability Methodology

An accountability report organizes aggregate statistics for undergraduate involvement in a certain activity type. An Activity models a person’s participation in some sort of trackable activity. The primary purpose of the Activity models is to compile and generate accountability statistics for the provost and the legislature. We currently focus on two metrics for undergraduates: public service and research.

How activities are stored

Activity is subclassed by a number of Activity varieties:

ActivityCourse:	for tracking an entire UW course for a particular quarter
ActivityProject:	for tracking an ongoing project, such as Students in Service, that lasts several weeks or even several quarters
ActivityEvent:	for tracking a one-time event, such as the MLK Day of Service, that has a more limited scope.

Sources

The following data sources are used to calculate accountability statistics:

• Course lists provided by departments, including
  course_branch, course_no, dept_abbrev, and section_id for each course along
  with a Y/N for each quarter that the course should be counted. (Note: if
  section_id was not specified, we automatically pull all sections
  for that course. In some cases this will result in duplicates but they will
  be removed during duplication processing).
• Individual student lists provided by departments
• Carlson Center service-learning participation (as tracked by EXPO) —
  only for the public service report
• Applicants and group members of applications whose Offering has an ActivityType defined (currently, this includes
  Mary Gates Research scholarship [research], Mary Gates leadership
  scholarship [public service], and the Undergraduate Research Symposium
  [research]).
  • For session-based events (e.g., UGR Symposium), session presenters (all
    group members) are counted.
  • For non-research-based award offerings (e.g., Mary Gates Leadership), only
    awarded applicants are counted.
  • For research-based award offerings (e.g., Mary Gates Research), all
    complete applications (mentor approved) are counted.

Intended Results

This system generates the following general types of statistics wherever
possible:

Student-quarters

The number of quarters a student spent working in the given activity. One
student for one quarter = 1 student-quarter. One student for three quarters
= 3 student-quarters. Three students for one quarter = 3 student-quarters.

Number of students

The total number of unique students who participated

Number of hours

A sum of all total number of hours that students participated

Coverage

The number/percentage of departments reporting information

Accountability Methodology

This section describes the methodology used to calculate accountability
statistics. All statistics are generated in groups based on the ActivityType of the given activity.

1. Raw Data AccumulationFor each data source, pull relevant records. Calculate number of students
   and hours for each record.
   1. Student participation is gathered for department
      course lists:
      1. Each ActivityCourse that meets the
         requested criteria (e.g., “year = 2009”) is validated against
         the SDB to verify that it is a valid course.
      2. For each course, the class enrollment is pulled, resulting in individual Student records.
      3. Students who withdrew, failed, or did not receive credit for the course are
         removed.
   2. Student participation is gathered from
      department individual participant lists.
   3. Student participation is gathered from
      EXPO’s service-learning courses (ONLY for public service report).
   4. Student participation is gathered from
      EXPO’s application system (ONLY matching ActivityTypes)

   Final result: hash with Students as keys and an activity array for each
   quarter, e.g.:

          Student_1_system_key   =>  {
            AutumnQuarter => [ActivityCourse, MaryGatesLeadershipApplication],
            SpringQuarter => [ServiceLearningCourse],
                           },
          Student_2_system_key   =>  {
            WinterQuarter => [ActivityEvent]
          }
          Student_3_system_key   => {
            :no_quarter => [ActivityProject]   # in rare cases, we don't know what quarters a project spanned
          }
          ...

2. Removal of non-undergraduates for each student in the data hash, check each quarter of activity to ensure
   that the student was enrolled as an undergraduate during that quarter.
   Non-undergraduate quarters are removed. To do this, we look at the
   student’s transcript record for that quarter.
   • If the quarter being evaluated is summer, the record is never removed, even
     if the student was not enrolled or not listed as an undergraduate for that
     quarter.
   • If there is no transcript record for that quarter and it is not summer, we
     remove it from the data hash.
   • If the “class” code for that quarter is higher than 5 and it is
     not summer, we remove it from the data hash.

   Notes:

   • For this report, “undergraduate” is any student with a class
     standing of 1, 2, 3, 4 or 5 (note that we do not include non-matriculated
     students but we do include “5th years”).
   • If removing a quarter from a student in the data hash results in that
     student having no more valid quarters, the student is completely removed
     from the data hash and will not count toward any totals.
   • Any :no_quarter listings are still analysed based on the first quarter of
     the academic year being evaluated.

   Final result: same as in step 1, but only with the students’ quarters
   that are valid.

3. Duplication ProcessingFor each student in the data hash, process each duplication rule (see rule
   definitions below). As the processing happens, records that are removed are
   saved in a duplicates array for later review.
   1. Same quarter, same course
   2. Same quarter, multiple courses
   3. Same quarter, multiple activities

   Final result: same as in step 2, but only with the activities that should
   be counted in the final tally.

4. Statistics Calculation
   1. Student-quarters: sum of the count of each student’s Quarter objects, plus any :no_quarter listings.
   2. Number of students: size of student data array.
   3. Number of hours: sum of number of hours per student; calculated differently
      based on the data source.
      1. Course list: number of credits the student
         earned for the course, multiplied by the constant
         COURSE_CREDIT_TO_HOURS_MULTIPLIER and the WEEKS_PER_QUARTER (10).
      2. Individual list: Hours are specified by department (as either hours per
         week or total hours per quarter)
      3. Carlson Center service-learning: 20 hours per quarter is used, as
         recommended by the Carlson Center (this is the minimum requirement by the
         CC)
      4. Application processes: use hours_per_week as specified by the student, if
         available. If hours_per_week is not available, we calculate the average
         number of hours for all students in the given activity type.
         • For award-based processes, multiply the calculated-hours-per-quarter times
           the number of requested award quarters
         • For event-based processes, the number of hours is only counted for the
           quarter during which the event occured.
5. Additional Reports
   • Department Sponsorship Breakdown: in what
     departments are these activities happening?
   • Department Cross-Pollination: e.g., how many
     BIO majors are doing research sponsored by medicine?
   • Demographic Analysis: majors, gender, class standing, and race/ethnicity of
     students participating.
   • Coverage: how many departments are represented/reported, broken down by
     college, discipline, and academic/admin.

Rules for duplicates

These rules define how we handle students who have multple records in a
given quarter. Note that activities are always
evaluated through the lens of a specific ActivityType, so these rules only apply to
duplicates within the same ActivityType.
For example, if a student participates in two research projects and one
public service course in the same quarter, the public service course is not
factored into the duplication processing of the two research projects.

Any time duplicates are removed, a Duplicates Report or Discrepancy Report
is generated to allow for manual inspection and override.

• Same quarter, same course: If a student is enrolled in the same course for
  the same course_no (e.g., they are enrolled in both a lecture section and a
  quiz section for the same course), only the section with the larger number
  of credits are counted.
• Same quarter, multiple courses: Students receiving credit for multiple
  courses in the same quarter are not de-duped, unless the faculty
  mentor/instructor is listed as the same for both course.
• Same quarter, multiple activities: Students who receive credit for a course
  and are also listed under an ActivityProject or ActivityEvent for the same quarter will only
  count the larger of the number of hours.