Purpose of the Project

The Task:  The project has formalized a component of the marketing research class for junior marketing majors at KSOM, bridging stated programmatic objectives mandated by AACSB (accrediting body for business programs at KSOM) and information literacy standards mandated by the Middle States accreditation standards.

The general activities related to this project have been covered by the faculty pair (Chattopadhyay and Moylan) in the past through a presentation by Prof. Moylan in Prof. Chattopadhyay’s classes with small assigned exercises.  The present project was to create a companion teaching-assessment module that can be linked to specific curriculum and program level outcomes.

Brief description of project

A course embedded assessment exercise was used.  The course chosen was MKT 361 Marketing Research.  This course is required for all Kania School of Management students with Marketing as their major.  All sections of the course are taught by Prof. Chattopadhyay.  The enrollment in the course is usually marketing majors only with one or two others. In this instance there was only one communications major in the class.

The students were provided with outdated information summaries on a number of countries from a source (CIA fact-book) available on the web.  Their task was to update the information to the most current available and provide updated citations for the updated data.

The students, in order to accomplish this task, had to first identify a set of information variables that can be used in an appropriate analytical model (covered in MKT 351 Introduction to Marketing) and then acquire valid information from secondary sources (topics covered in MKT 361 Marketing Research, prior to start of project exercise will include reliability and validity of data) to populate the model parameters.  The assessment assignment is provided in PDF-Appendix A.
 
Dr. Chattopadhyay covered the topics of “market potential modeling,” and “decision modeling” prior to beginning of exercise.  He provided an example using old data and a product market different from the one used in the actual exercise.

Before the assignment was handed out to the student, Prof. Moylan lectured on library resources for identifying and validating information related to estimating country-market potential.

Both professors were available to assist students at specially designated and pre-announced hours at the library.

A rubric to assess the learning outcomes was identified and used to report the results of this exercise.  The Rubric used to assess performance is provided in PDF-Appendix B.

Results:

The assignment was posted on Blackboard on February 27, 2006 and the students submitted the completed assignment on March 6, 2006.  For the purpose of computing the grade for the course, the assignment was given full credit if completed and submitted in time.  The submissions were further used for assessment purpose.

46 usable assignments were obtained form the two sections with total enrollment of 49.

The rubric was applied to the assignments in the following manner:  The assignments were first numbered 1-46 without any specific order, so that the names of the students would be masked.  Then ten passes were made through the assignments, applying the rubric to rate the information on each of the 10 variables that were specified for the 12 target countries:

1. population growth rate
2. GDP in USD
3. GDP growth rate
4. Country Risk
5. Balance of Payments
6. BigMac Index
7. BigMac Index
8. Poverty Index
9. Volume of Trade with US
10. Corruption Index

Each item was scored on a 1-4 scale where 1=unacceptable, 2=acceptable, 3=very good, and 4=exceptional based on the scoring rubric adapted.

The raw scoring data in an Excel file is provided in PDF- Appendix C.

The data was then saved in a SPSS data file.  Additional variables were created:

1. Average score across all data collected
2. Average score associated with standard economic variables
3. Average score associated with special economic indices

The SPSS dataset can be seen in PDF- Appendix D (data view) and PDF-Appendix E (variable view)

Findings:

Three assessment scores were examined: 
Average Score = Average score on all 10 data items rounded to nearest integer
Average Score on Economic Data = Avg. score on the pop. growth, GDP, GDP growth, balance of payments, volume of trade with US
Average Score on Special Indices = Avg. score on the country risk, BigMac Index, Human Development Index, Poverty Index and Corruption Index.

The scores data can be treated as interval data (the values are at equal appearing intervals), so means. Modes and medians on these variables are presented in Table 1.

Table 1 Data Descriptive

Table 2 Overall Assessment of Data Retrieved

It appears that the students assessed were fairly high performers on the objective/standard assessed.  43.5% of the students scored at the exceptional level and 52.2% scored as “very good.”  Only 4.3% came in at the acceptable level and there were no “unacceptable” scores.

Table 3 Assessment of Economic Data Retrieved

For retrieving data on standard economic variables from commonly used secondary sources such as government published statistics, and data put out by multilateral aid agencies such as UNDP, World Bank and the like, the participants were stronger than believed.  A full 56.5% of the students demonstrated their competence at the exceptional level, and 2.2% came in as “acceptable,” the rest 41.3% being rated as “very good.” There were no “unacceptable” scores.

Table 4 Assessment of Special Indice Data Retrieved

For retrieving data on specialized indexes that are regularly published by special interest organizations, the participants scored marginally lower.  41.3% of the participants demonstrated their competence at the exceptional level, and 4.3% came in as “acceptable,” the rest 54.3% being rated as “very good.” There were no “unacceptable” scores for this measure, either.

Table 5 Correlation of Assessment Scores on Economic Data and Special Indices

Upon further exploratory analysis, it was found that the information literacy scores when assessed for standard economic data were highly correlated (0.831) with those for data on special purpose indices.  A paired sample t-test performed on the scores of the two types of data retrieved was found to be significant, as seen in Table 6.

Table 6 Information Literacy Variation over type of data retrieved

Implications of the Assessment Exercise
The results of this assessment indicate that overall, for the population tested, the KSOM curriculum has been successful in ensuring that the students are able to meet the following standard which is an integral part of the student outcomes indicated in its strategic plan:

Standard Three: The information literate student, individually or as a member of a group, uses information effectively to accomplish a specific purpose
   Performance Indicators:  

1. The information literate student articulates and applies initial criteria for evaluating both the information and its sources.
   Outcomes Include:
• Examines and compares information from various sources in order to evaluate reliability, validity, accuracy, authority, timeliness, and point of view or bias

While overall, the standards are being met successfully, in the spirit of continuous improvement, as required by AACSB, further exploration of the data was undertaken.  While this was not planned in advance, the data offered the chance to look at student competence in retrieving data from traditional economic sources versus retrieving from non-traditional sources (in this case, various special purpose indices) that are now being put out which focus on broader conceptual measures of economic environment.  These data, being mostly generated by special interest organizations have to be used carefully as the mechanisms for their generation and computation are not as widely available.  However, on the positive side, these indices provide more comprehensive and holistic measures that are often more conceptually appealing to analysts.

Thus the small but statistically significant difference in information literacy scores for these two classes of sources have potential for effecting improvements in information literacy of the students of KSOM.

Recommended Classroom Strategy:
In the various courses for Economics, Marketing, Management, Sustainability that are taught to KSOM students, it will be beneficial if students are made aware of various think-tanks, public interest research groups (PIRGs), special interest groups, and non-governmental agencies that gather and publish data on holistic measures and indices that are increasingly being used as performance indicators that contribute to policy, strategy and tactics in the business world.  It is recommended that wherever possible, a portion of a class period or more be set aside to discuss the value of the information content provided by these sources and how to make reasoned conclusions about their reliability and validity, in order to include them in decision-making.