Abstracts
Abstract
Measurement error arises from many sources in educational assessment. It is important to estimate the importance of this error, and, if appropriate, to seek ways to reduce it. Generalizability theory represents a powerful tool in this sense, allowing identifiable error contributions to be separately quantified, and measurement error to be estimated and even predicted in response to possible changes in the measurement procedure. The paper offers examples of generalizability analysis of numeracy attainment data deriving from the Scottish Survey of Achievement, with the aim of illustrating the versatility of the methodology for error estimation and prediction in this type of sample-based programme.
Keywords:
- Generalizability theory,
- national assessment,
- attainment surveys,
- numeracy assessment,
- test reliability,
- domain sampling,
- matrix sampling
Résumé
L’erreur de mesure découle de nombreuses sources dans l’évaluation en éducation. Il est important d’estimer l’ampleur de cette erreur et, si c’est le cas, de chercher les moyens de la réduire. La théorie de la généralisabilité représente dans ce sens un outil puissant qui permet d’identifier les sources de l’erreur et de les quantifier séparément, d’estimer l’erreur de mesure et même de prédire la réponse à d’éventuels changements dans la procédure de mesure. Cet article offre des exemples d’application de l’analyse de la généralisabilité sur des données pour le suivi des acquis des notions de calcul, données provenant de l’enquête écossaise sur la réussite, dans le but d’illustrer la polyvalence de la méthodologie d’estimation et de prévision de l’erreur dans ce programme d’évaluation basé sur un échantillonage.
Mots-clés :
- Théorie de la généralisabilité,
- évaluation nationale,
- enquêtes pour le suivi des acquis,
- évaluation des notions de calcul,
- fiabilité d’un test,
- échantillonnage par domaine,
- échantillonnage matriciel
Resumo
Na avaliação em educação, o erro de medida decorre de numerosas fontes. É importante calcular a amplitude deste erro e, se for o caso, procurar meios para a reduzir. A teoria da generalizabilidade representa, neste sentido, um instru mento poderoso que permite identificar as fontes do erro e quantificá-las separadamente, calcular o erro de medida e mesmo prever a resposta a eventuais mudanças nos procedimentos de medida. Este artigo fornece exemplos de aplicação da análise da generalizabilidade sobre os dados das aprendizagens adquiridas de noções de cálculo, dados provenientes do Estudo Escocês sobre o Sucesso, com o objectivo de ilustrar a versatilidade da metodologia de cálculo e previsão do erro neste programa de avaliação baseado numa amostra.
Palavras chaves:
- Teoria da generalizabilidade,
- avaliação nacional,
- estudos das aprendizagens adquiridas,
- avaliação das noções de cálculo,
- fiabilidade de um teste,
- amostragem por domínio,
- amostragem matricial
Appendices
References
- Brennan, R.L. (1992). Elements of generalizability theory (2nd edition). Iowa City: ACT Publications (First edition: 1983).
- Brennan, R.L. (2001). Generalizability theory. New York: Springer-Verlag.
- Cardinet, J., & Tourneur, Y. (1985). Assurer la mesure. Berne: Peter Lang.
- Cardinet, J., Tourneur, Y., & Allal, L. (1976). The symmetry of generalizability theory: applications to educational measurement. Journal of Educational Measurement, 13(2), 119-135.
- Cardinet, J., Tourneur, Y., & Allal, L. (1981, 1982). Extension of generalizability theory and its applications in educational measurement. Journal of Educational Measurement, 18(4), 183-204, and Errata, 19(4), 331-332.
- Condie, R., Robertson, I.J., & Napuk, A. (2003). The assessment of achievement programme. In T.G. K. Bryce & W.M. Humes (Eds), Scottish Education (pp. 766- 776).Edinburgh: Edinburgh University Press.
- Cronbach, L. J. (1951). Coefficient Alpha and the internal structure of tests. Psychometrika, 16, 297-334.
- Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
- Hayward, E.L. (2007). Curriculum, pedagogies and assessment in Scotland: the quest for social justice. ‘Ah kent yir faither’. Assessment in Education, 14(2), 251-268.
- Hogan, T.P, Benjamin, A., & Brezinski, K.L. (2000, 2003). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523-531.
- Johnson, S. (2003). Une application de la théorie de la généralisabilité à la planification des enquêtes sur les acquisitions des élèves. In J. Cardinet (Ed.), Special Issue: Generalizability Theory. Mesure et évaluation en éducation, 26(1-2), 37-59.
- Johnson, S., & Bell, J. (1985). Evaluating and predicting survey efficiency using generalizability theory. Journal of Educational Measurement, 22, 107-119.
- Scottish Executive (2004). Scottish Index of Multiple Deprivation 2004: Summary technical report. Edinburgh: Scottish Executive.
- Scottish Government (2006). The 2005 Scottish Survey of Achievement (SSA): English language and core skills. [http://www.scotland.gov.uk/Publications/2006/06/29141936/0]
- Scottish Government (2007). Scottish Survey of Achievement: 2006 Social subjects (enquiry skills) and core Skills. [www.scotland.gov.uk/Publications/2007/08/15104710/0]
- Shavelson, R., & Webb, N. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
- Snijders, T.A.B, & Bosker, R.J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modelling. London: Sage Publications.
- SOED (1991). National guidelines: Mathematics 5-14. Edinburgh: Scottish Office Education Department. [http://www.ltscotland.org.uk/5to14/guidelines/mathematics.asp]
- SOEID (1999). National guidelines: Mathematics 5-14 Level F. Edinburgh: Scottish Office Education and Industry Department. [http://www.ltscotland.org.uk/5to14/guidelines/mathematics.asp]
- Thompson, B. (2003). A brief introduction to generalizability theory. In B. Thompson (Ed.), Score reliability (pp. 43-58). Thousand Oaks, CA: Sage publications.