institut sur la gouvernance d’organisations privées et...
TRANSCRIPT
Institut sur la gouvernance drsquoorganisations priveacutees et publiques
Collection Gouvernance des institutions universitaires
Les travaux de cette collection sont dirigeacutes par Jean-Marie Toulouse PhD professeur titulaire HEC Montreacuteal
Document 4
RAPPORT DE RECHERCHE SUR LES INDICATEURS
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Roland Proulx est professeur honoraire de lrsquoUniversiteacute de Montreacuteal ougrave il a eacuteteacute Directeur de la planification de 1984 agrave 2003 Il a eacuteteacute membre du bureau des Directeurs de la Society for College and University Planning (1999‐2004) Il est actuellement consultant en planification institutionnelle et veille strateacutegique membre de lrsquoInternational Ranking Expert Group (Unesco‐Cepes) et partner de Strategic Initiative management consultants (USA)
Institut sur la gouvernance drsquoorganisations priveacutees et publiques 2008 3000 chemin de la Cocircte‐Sainte‐Catherine Bureau 4311 Montreacuteal (Queacutebec) H3T 2A7 Teacuteleacutephone (514) 340‐6398 Site Web wwwigopporg Deacutepocirct leacutegal ndash Mai 2008 Bibliothegraveque nationale du Queacutebec Bibliothegraveque nationale du Canada ISBN 978‐2‐9810410‐5‐0 Ce document est disponible gratuitement sur le site wwwigopporg La reproduction drsquoextraits est autoriseacutee agrave des fins non commerciales avec mention de la source Toute reproduction partielle doit ecirctre fidegravele au texte utiliseacute
wwwigopporg 2
PREMIEgraveRE PARTIE
INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DrsquoEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
Preacutesentation
Inventaire des indicateurs
Annexes quelques exemples de listes drsquoindicateurs
On peut reacutesister agrave lrsquoinvasion drsquoune armeacutee mais pas agrave une ideacutee - Victor Hugo Histoire drsquoun crime
wwwigopporg 3
La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees
soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine
drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et
de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des
programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐
vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de
lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique
emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave
leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George
Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in
Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐
ameacutericaines
Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des
indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs
composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus
agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent
en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key
performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de
la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent
compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant
qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les
universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les
demandes de reddition de compte formuleacutees entre autres par les gouvernements les
Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant
national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs
repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories
inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash
processus ndash output ndash reacutesultat
wwwigopporg 4
Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les
gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre
varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp
alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux
favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour
caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les
sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes
comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs
preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants
comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes
On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes
autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries
drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une
innovation)
Eacutetudiants
Enseignants
Personnels
Activiteacutes de recherche
Internationalisation
Inteacutegration des technologies de linformation
Bibliothegraveques
Revenus et deacutepenses
Espaces
Environnement
Reacuteputation
Planification et gestion strateacutegiques
Eacuteducation permanente
wwwigopporg 5
wwwigopporg
Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus
Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes
courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et
de reddition de comptes De plus dans un contexte de mondialisation et
drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de
retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)
et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix
classements nationaux classements internationaux
universiteacutes canadiennes
et MEQ
USNews and
Report ampThe
Center
Macleans CHEDAAD GUARDIAN ASIAWEEKDEST
AUSTRALIE amp
MELBOURNE REPUBLICA
QS-TIMES SHANHAI TAIWAN
WEB
1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs
6
1INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EacutetudiantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIEampMel
bourneREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ
inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica
inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek
provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input
inputdiversiteacute ethnique input US NEWS Guardian
Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10
cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica
Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans
taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne
Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian
RP gouvernance - 2008-06-23
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
Institut sur la gouvernance drsquoorganisations priveacutees et publiques 2008 3000 chemin de la Cocircte‐Sainte‐Catherine Bureau 4311 Montreacuteal (Queacutebec) H3T 2A7 Teacuteleacutephone (514) 340‐6398 Site Web wwwigopporg Deacutepocirct leacutegal ndash Mai 2008 Bibliothegraveque nationale du Queacutebec Bibliothegraveque nationale du Canada ISBN 978‐2‐9810410‐5‐0 Ce document est disponible gratuitement sur le site wwwigopporg La reproduction drsquoextraits est autoriseacutee agrave des fins non commerciales avec mention de la source Toute reproduction partielle doit ecirctre fidegravele au texte utiliseacute
wwwigopporg 2
PREMIEgraveRE PARTIE
INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DrsquoEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
Preacutesentation
Inventaire des indicateurs
Annexes quelques exemples de listes drsquoindicateurs
On peut reacutesister agrave lrsquoinvasion drsquoune armeacutee mais pas agrave une ideacutee - Victor Hugo Histoire drsquoun crime
wwwigopporg 3
La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees
soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine
drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et
de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des
programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐
vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de
lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique
emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave
leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George
Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in
Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐
ameacutericaines
Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des
indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs
composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus
agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent
en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key
performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de
la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent
compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant
qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les
universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les
demandes de reddition de compte formuleacutees entre autres par les gouvernements les
Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant
national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs
repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories
inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash
processus ndash output ndash reacutesultat
wwwigopporg 4
Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les
gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre
varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp
alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux
favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour
caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les
sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes
comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs
preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants
comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes
On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes
autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries
drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une
innovation)
Eacutetudiants
Enseignants
Personnels
Activiteacutes de recherche
Internationalisation
Inteacutegration des technologies de linformation
Bibliothegraveques
Revenus et deacutepenses
Espaces
Environnement
Reacuteputation
Planification et gestion strateacutegiques
Eacuteducation permanente
wwwigopporg 5
wwwigopporg
Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus
Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes
courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et
de reddition de comptes De plus dans un contexte de mondialisation et
drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de
retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)
et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix
classements nationaux classements internationaux
universiteacutes canadiennes
et MEQ
USNews and
Report ampThe
Center
Macleans CHEDAAD GUARDIAN ASIAWEEKDEST
AUSTRALIE amp
MELBOURNE REPUBLICA
QS-TIMES SHANHAI TAIWAN
WEB
1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs
6
1INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EacutetudiantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIEampMel
bourneREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ
inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica
inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek
provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input
inputdiversiteacute ethnique input US NEWS Guardian
Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10
cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica
Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans
taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne
Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian
RP gouvernance - 2008-06-23
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
PREMIEgraveRE PARTIE
INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DrsquoEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
Preacutesentation
Inventaire des indicateurs
Annexes quelques exemples de listes drsquoindicateurs
On peut reacutesister agrave lrsquoinvasion drsquoune armeacutee mais pas agrave une ideacutee - Victor Hugo Histoire drsquoun crime
wwwigopporg 3
La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees
soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine
drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et
de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des
programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐
vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de
lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique
emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave
leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George
Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in
Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐
ameacutericaines
Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des
indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs
composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus
agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent
en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key
performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de
la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent
compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant
qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les
universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les
demandes de reddition de compte formuleacutees entre autres par les gouvernements les
Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant
national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs
repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories
inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash
processus ndash output ndash reacutesultat
wwwigopporg 4
Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les
gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre
varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp
alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux
favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour
caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les
sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes
comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs
preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants
comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes
On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes
autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries
drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une
innovation)
Eacutetudiants
Enseignants
Personnels
Activiteacutes de recherche
Internationalisation
Inteacutegration des technologies de linformation
Bibliothegraveques
Revenus et deacutepenses
Espaces
Environnement
Reacuteputation
Planification et gestion strateacutegiques
Eacuteducation permanente
wwwigopporg 5
wwwigopporg
Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus
Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes
courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et
de reddition de comptes De plus dans un contexte de mondialisation et
drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de
retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)
et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix
classements nationaux classements internationaux
universiteacutes canadiennes
et MEQ
USNews and
Report ampThe
Center
Macleans CHEDAAD GUARDIAN ASIAWEEKDEST
AUSTRALIE amp
MELBOURNE REPUBLICA
QS-TIMES SHANHAI TAIWAN
WEB
1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs
6
1INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EacutetudiantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIEampMel
bourneREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ
inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica
inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek
provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input
inputdiversiteacute ethnique input US NEWS Guardian
Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10
cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica
Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans
taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne
Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian
RP gouvernance - 2008-06-23
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
La production drsquoindicateurs et leur application aux universiteacutes remontent agrave la fin des anneacutees
soixante Les premiegraveres listes apparaissent en 1968 soit il y a deacutejagrave une cinquantaine
drsquoanneacutees Les premiers indicateurs se preacutesentent sous la forme de donneacutees statistiques et
de caracteacuteristiques institutionnelles portant sur le nombre drsquoeacutetudiants la qualiteacute des
programmes et les problegravemes financiers que rencontrent les universiteacutes Les anneacutees quatre‐
vingt teacutemoignent drsquoun changement majeur lrsquointroduction dans le domaine de
lrsquoenseignement supeacuterieur des concepts et des techniques de planification strateacutegique
emprunteacutes au monde des affaires incite au deacuteveloppement drsquoautres types drsquoindicateurs et agrave
leur application agrave lrsquoensemble des domaines drsquoactiviteacutes universitaires Le livre de George
Keller publieacute en 1983 sous le titre de Academic Strategy The Management Revolution in
Higher Educations est significatif du virage majeur qui srsquoopegravere dans les universiteacutes nord‐
ameacutericaines
Nous assistons au cours de la mecircme peacuteriode agrave la naissance drsquoune veacuteritable typologie des
indicateurs Aux indicateurs simples et descriptifs viennent srsquoajouter des indicateurs
composeacutes (ratios pourcentage) les deux cateacutegories drsquoindicateurs se reacutefeacuterant de plus en plus
agrave des normes agrave des standards agrave des objectifs et agrave des cibles Les critegraveres retenus eacutevoluent
en direction drsquoindicateurs de rendement de performance de suivi La notion de laquo key
performance indicators (KPIs) raquo se retrouve au cours des derniegraveres anneacutees dans le titre de
la plupart des publications et dans la production drsquoindicateurs Plusieurs facteurs rendent
compte de cette eacutevolution Il faut retenir agrave cet eacutegard lrsquointeacutegration des indicateurs tant
qualitatifs que quantitatifs au processus de planification strateacutegique qursquoadoptent de plus les
universiteacutes lrsquoeacutevaluation par les pairs des faculteacutes et des deacutepartements des universiteacutes les
demandes de reddition de compte formuleacutees entre autres par les gouvernements les
Conseils drsquoadministration les laquo stakeholders raquo et le positionnement strateacutegique tant
national qursquointernational des universiteacutes Il est devenu eacutevident pour tous que les indicateurs
repreacutesentent un certain modegravele de production qui srsquoarticule autour de quatre cateacutegories
inter‐relieacutes de mesures que lrsquoon retrouve dans la plupart des listes et drsquoindicateurs input ndash
processus ndash output ndash reacutesultat
wwwigopporg 4
Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les
gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre
varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp
alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux
favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour
caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les
sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes
comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs
preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants
comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes
On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes
autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries
drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une
innovation)
Eacutetudiants
Enseignants
Personnels
Activiteacutes de recherche
Internationalisation
Inteacutegration des technologies de linformation
Bibliothegraveques
Revenus et deacutepenses
Espaces
Environnement
Reacuteputation
Planification et gestion strateacutegiques
Eacuteducation permanente
wwwigopporg 5
wwwigopporg
Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus
Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes
courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et
de reddition de comptes De plus dans un contexte de mondialisation et
drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de
retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)
et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix
classements nationaux classements internationaux
universiteacutes canadiennes
et MEQ
USNews and
Report ampThe
Center
Macleans CHEDAAD GUARDIAN ASIAWEEKDEST
AUSTRALIE amp
MELBOURNE REPUBLICA
QS-TIMES SHANHAI TAIWAN
WEB
1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs
6
1INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EacutetudiantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIEampMel
bourneREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ
inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica
inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek
provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input
inputdiversiteacute ethnique input US NEWS Guardian
Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10
cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica
Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans
taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne
Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian
RP gouvernance - 2008-06-23
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
Les indicateurs font partie deacutesormais des outils indispensables auxquels ont recours les
gouvernements la communauteacute universitaire les eacutetudiants et leurs parents Leur nombre
varie entre dix et 300 indicateurs Bon nombre drsquoauteurs (Barbara Taylor (1993) Gaither amp
alii (1994) drsquoorganismes (COU) de producteurs de classements nationaux et internationaux
favorisent des listes courtes de 10 agrave 20 indicateurs ces listes courtes sont suffisantes pour
caracteacuteriser les activiteacutes drsquoune universiteacute repreacutesentent des indicateurs courants dont les
sources sont accessibles et permettent le repeacuterage (benchmarking) des universiteacutes
comparables En produisant des listes longues de 100 agrave 300 indicateurs les producteurs
preacutesentent plutocirct un reacutepertoire ou une banque drsquoindicateurs parmi lesquels les requeacuterants
comme les utilisateurs peuvent piger selon les objectifs poursuivis et les inteacuterecircts privileacutegieacutes
On trouvera dans les pages qui suivent un inventaire de quelque 175 indicateurs regroupeacutes
autour de 13 chefs de division ou domaines drsquoactiviteacutes (parmi lesquels deux seacuteries
drsquoindicateurs ndash environnement et planification et gestion strateacutegique constituent une
innovation)
Eacutetudiants
Enseignants
Personnels
Activiteacutes de recherche
Internationalisation
Inteacutegration des technologies de linformation
Bibliothegraveques
Revenus et deacutepenses
Espaces
Environnement
Reacuteputation
Planification et gestion strateacutegiques
Eacuteducation permanente
wwwigopporg 5
wwwigopporg
Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus
Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes
courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et
de reddition de comptes De plus dans un contexte de mondialisation et
drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de
retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)
et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix
classements nationaux classements internationaux
universiteacutes canadiennes
et MEQ
USNews and
Report ampThe
Center
Macleans CHEDAAD GUARDIAN ASIAWEEKDEST
AUSTRALIE amp
MELBOURNE REPUBLICA
QS-TIMES SHANHAI TAIWAN
WEB
1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs
6
1INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EacutetudiantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIEampMel
bourneREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ
inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica
inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek
provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input
inputdiversiteacute ethnique input US NEWS Guardian
Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10
cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica
Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans
taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne
Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian
RP gouvernance - 2008-06-23
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
wwwigopporg
Tous les indicateurs des listes longues et tous les domaines drsquoactiviteacutes nrsquoont pas eacuteteacute retenus
Il srsquoagit drsquoun inventaire des indicateurs courants tels que recenseacutes dans plusieurs listes
courtes et longues et utiliseacutes agrave des fins de planification drsquoeacutevaluation de la performance et
de reddition de comptes De plus dans un contexte de mondialisation et
drsquointernationalisation il est devenu important et pertinent drsquoun point de vue strateacutegique de
retenir les indicateurs utiliseacutes dans les principaux classements nationaux (9 sur 20 possibles)
et internationaux (4) 1 Lrsquoencadreacute qui suit reacutesume ce choix
classements nationaux classements internationaux
universiteacutes canadiennes
et MEQ
USNews and
Report ampThe
Center
Macleans CHEDAAD GUARDIAN ASIAWEEKDEST
AUSTRALIE amp
MELBOURNE REPUBLICA
QS-TIMES SHANHAI TAIWAN
WEB
1 On trouvera en fin de document lrsquoensemble des reacutefeacuterences consulteacutees ainsi qursquoun annexe preacutesentant quelques exemples de listes drsquoindicateurs
6
1INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EacutetudiantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIEampMel
bourneREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ
inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica
inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek
provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input
inputdiversiteacute ethnique input US NEWS Guardian
Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10
cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica
Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans
taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne
Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian
RP gouvernance - 2008-06-23
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
1INVENTAIRE DES INDICATEURS COURANTS
Agrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EacutetudiantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIEampMel
bourneREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre deacutetudiantsNombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input MEQ G10 DETYA THESARWUproportion deacutetudiantes input DETYA des eacutetudiants agrave temps partiel input AOTQU CHEDAAD Republicainscriptions au 1er cycle agrave temps plein input MEQ CHEDAAD inscriptions au 2e cycle input MEQ inscriptions au 3e cycle input MEQ eacutetudiants aux 2e et 3e cycles input G10 DETYA eacutetudiants internationaux input TU THES des eacutetudiants internationaux input Macleans CHEDAAD THES des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input MEQ stagiaires postdoctoraux input MEQ reacutesidents et reacutesidentes en meacutedecine input MEQ
inputrecrutement et admission inputdemandes dadmission aux programmes reacuteguliers input OTQ CHEDAAD offres dadmissiondemandes dadmission i t CHEDAAD offres dadmissiondemandes dadmission input CHEDAAD inscriptionsdemandes dadmission input T CHEDAAD Asiaweek Republica
inputdossier scolaire des nouveaux eacutetudiants (bulletin Cote R) input US NEWS Macleans Asiaweek
provenance des eacutetudiants input Macleanspopulation cible inputreacutegion meacutetropolitaine inputQueacutebec inputCanada inputInternational input
inputdiversiteacute ethnique input US NEWS Guardian
Bourses accordeacutees aux eacutetudiants - Financement moyen input TMEQpremier cycle input Macleans Republicadeuxiegraveme cycle input G10troisiegraveme cycle input G10
cheminement des eacutetudiants (temps complet) perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output QT MEQ US NEWS Macleans Republica diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output G10 MEQ US NEWS Macleans Melbourne Republica dilplomationmaicirctrise apregraves 4 ans output G10 MEQ US NEWS Macleans Republica diplomationdoctorat apregraves 6 ans output G10 MEQ US NEWS Macleans Republica
Deacutepenses de fonctionnement par eacutetudiant input CAUBOG10 US NEWS Macleans Guardian Asia weekFrais de scolariteacute input G10frais affeacuterents inputDeacutepenses pour les Services aux eacutetudiants input Macleans
taux demployabiliteacute apregraves 1 an outcome Ontario CHEDAAD Guardian de retour aux eacutetudes outcome Melbourne
Valeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome Guardian
RP gouvernance - 2008-06-23
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
2INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EnseignantsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre denseignants Nombre total denseignants inputNombre de professeurs reacuteguliers input G10 MEQStatCan USNEWS Melbourne de professeures input OQ MelbourneNombre de chercheurs inputNombre de professeurs avec diplocircme international THESNombre de professeurs de clinique eacutequivalents temps complet input G10StatCanNombre de chargeacutes de cours et autres personnels eacutequivalents temps complet input USNEWSNombre dauxiliaires en eacutequivalence temps complet input
Caracteacuteristiques des professeurs reacuteguliers input des professeurs et chercheurs avec PhD (ou grade terminal) input USNEWS Macleansstructure dage du corps professoral input Asiaweek Melbourne Republicaacircge moyen des professeurs reacuteguliers inputacircge moyen des professeurs adjoints inputacircge moyen des professeurs agreacutegeacutes inputacircge moyen des professeurs titulaires input
reacutemuneacuteration moyenne des professeurs reacuteguliers input Q T U U S NEWS Asiaweek Melbournereacutemuneacuteration moyenne des professeurs reacuteguliers input QTU USNEWS Asiaweek Melbournedes professeurs adjoints inputdes professeurs agreacutegeacutes inputdes professeurs titulaire input
Activiteacutes denseignement inputnombre deacutetudiants EETC Professeur reacutegulier input G10 MEQ USNEWS Guardian Asiaweek Melbourne Republica THESnombre deacutetudiants EETC Professeur reacutegulier en sciences sociales input Melbournenombre deacutetudiants EETC Professeur reacutegulier en sciences input Melbournenombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input
Nombre de cours dispenseacutes par lInstitution (ANNUEL) input Republica cours assumeacutes par les professeurs reacuteguliers input MEQ USNEWS Macleans Republica cours assumeacutes par les chargeacutes de cours eacutequivalents temps complet input MEQTaille des groupe cours input MEQ USNEWS Macleans Guardian
RP gouvernance - 2008-06-23
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
3INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
PersonnelsUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Le personnel de soutientotal de tous les personnels reacuteguliers acadeacutemiques et non acd agrave temps complet inputnombre des personnels de soutien (staff) input G10 MelbourneDeacutepenses des personnel non-enseignant input MEQnombre des personnels de soutien (staff)- secteur enseignement inputnombre des personnels de soutien (staff) - secteur soutien acadeacutemique inputnombre des personnels de soutien (staff) - secteur administration inputnombre des personnels de soutien (staff) - secteur service aux eacutetudiants inputnombre des personnels de soutien (staff) - entreprises auxiliaires input
rectoratpersonnel dencadrement non acadeacutemique et cadres acadeacutemiques inputratio sur lensemble des personnels acadeacutemiques et non acadeacutemiques inputeacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input G10 Guardianpersonnel de soutien professeur reacutegulier input G10
RP gouvernance - 2008-06-23
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
4INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Activiteacutes de recherche USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Revenus de recherche de professeurs et chercheurs ayant fait une demande de subvention processus de professeurs et chercheurs ayant obtenu une subvention inputRevenus totaux de recherche (subventions et contrats) input CAUBO G10 The Center Asiaweek Melbournepart de la recherche contractuelle input MEQSubventions publiques input Macleans Melbournerevenus de recherche obtenus des conseils de recherche feacutedeacuteraux input AOQTU The Center Macleans Melbournerevenus moyens de recherche par professeur et chercheur input InfosourceMEQ des deacutepenses de recherchebudget total des deacutepenses input TNombre de chaires et de centre dexcellence output AFondation canadienne pour linnovation (FCI) input G10 -nombre de projets output G10 -valeur $ output G10
nombre de programmes de maicirctrise de recherche input The Center Asiaweeknombre de programmes de doctorat input The Center Asiaweek
Publications et prix outputNombre de publications output Infosource Asiaweek Melbourne
Shanghai Leiden Taiwan
Nombre de publications par professeur et chercheur output Infosource CHEDAAD ShanghaiPublications dans Nature and Science output TaiwanNombre total de citations output InfosourceT Asiaweek MelbourneNombre de citations par professeur et chercheur output InfosourceT Melbourne THESNombre de citations par publication output T CHEDAAD LeidenHighly cited researchers dans 21 secteurs disciplinaires
output
ShanghaiTaiwan Leiden
Impact des publications selon les secteurs disciplinaires output Infosource TaiwanTHES Shanghai
Domaines disciplinaires dexcellence inputTaiwanShanghai THES
Prix et meacutedailles de niveau national et international output G10 The Center Macleans Melbourne Republica ShanghaiNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur output CHEDAADnombre de brevets et de licences output G10 CHEDAADnombre de spin-off companies G10
RP gouvernance - 2008-06-23
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
5INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
InternationalisationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
deacutepenses pour le secteur internationalisation input MEQnombre deacutetudiants en stage agrave leacutetrangernombre deacutetudiants temps complet input tes un cannombre deacutetudiants internationauxnombre deacutetudiants temps complet input tes un can Melbourne THESnombre deacutetudiants en cotutellesnombre deacutetudiant des cycles supeacuterieurs input CHEDAADnombre de professeurs inviteacutes internationaux processus tes un can CHEDAADprofesseurs inviteacutes agrave leacutetranger processus CHEDAADnombre deacutetudiants posdoctoraux processus tes un can CHEDAADproportion de cours de langues eacutetrangegraveres processusnombre de programmes deacutechange processus tes un can CHEDAADEntentes de partenariat outputPrix internationaux output G10 The Center Macleans Melbourne Republica Shanghai
RP gouvernance - 2008-06-23
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
6INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Inteacutegration des technologies de linformation USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Deacutepenses totales consacreacutees aux TIC input MEQpersonnel techniqueenseignant reacutegulier inputpersonnel techniqueeacutetudiant en eacutequivalence temps complet inputExistence dun reacuteseau informatique (accessible agrave linterne et agrave lexterne) outputaccegraves au reacuteseau avec fils et sans fils processusaccegraves aux ressources documentaires des bibliothegraveques outputService de courriel interne et externe en mode WEB outputNombre dordinateurs par professeur et par eacutetudiant outputNombre de cours en ligne processusnombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) outputAccegraves des professeurs aux banque de donneacutees et aux logiciels processus
Site WEB (eacutevaluation du site WEB) WEBOMETRICSSize (nombre de pages recenseacutees) outcomevisibilitty (consultation observeacutee selon le nobre de lien externe unique) outcomeRich files (intensiteacute de lactiviteacute acadeacutemique) outcomeScholar (activiteacute de recherche publications et citations) outcome
RP gouvernance - 2008-06-23
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
7INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
BibliothegravequesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre total de volumes et de documents audiovisuels et eacutelectroniques input G10 ARLacquisition de volumes (au cours de lanneacutee) input ARL Macleansnombre total de volumes eacutetudiants en eacutequivalence temps complet input G10 ARL Macleans Deacutepenses des bibliothegraveques input MEQ Macleans AsiaweekDeacutepenses des bibliothegravequesbudget de fonctionnement input ARLDeacutepenses des bibliothegravequeseacutetudiants en eacutequivalence temps complet input G10 ARLPersonnels reacuteguliers agrave temps complet input G10 ARLClassement outcome ARL
RP gouvernance - 2008-06-23
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
8INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Revenus et deacutepenses USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Produits et revenusSubvention du ministegravere de lEacuteducation du Queacutebec input MEQ subvMEQrevenus de fonctionnement input G10 MacleansDroits de scolariteacute input G10 droits de scolariteacute revenus de fonctionnement input G10autres produits input G10Total des revenus de fonctionnement input G10 MEQ Guardian revenus de fonctionnement eacutetudiants en eacutequivalence temps complet input CAUBO USNews MacleansRevenus de recherche input G10encaissement par nature (dons immobilisations chaires bourses) input G10 fonds de dotation input G10 CenterContribution des alumni aux encaissements et fonds de dotation input USNews Macleans fonds de dotation eacutetudiants en eacutequivalence temps complet input G10Total des produits G10
Charges et deacutepenses G10salaire des professeurs input G10 MEQsalaire des chargeacutes de cours input G10salaires des auxiliaires input MEQsalaires des auxiliaires input MEQsalaire des personnels (staff) input G10autres deacutepenses input G10total des deacutepenses input G10 deacutepenses denseignement deacutepenses totales de fonctionnement input UBC MEQ
Chargesproduits processusEacutequilibre budgeacutetaire revenus et deacutepenses de fonctionnement processus MEQ
RP gouvernance - 2008-06-23
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
9INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
EspacesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Nombre de megravetres carreacutes bruts input G10MCB eacutetudiants en eacutequivalence temps complet input G10megravetres carreacutes nets alloueacutes input MEQNombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input Tbesoin despace en megravetres carreacutes nets inputExistence dun plan directeur des espaces inputBudget preacutevu au plan directeur des espaces input TBudget alloueacute aux espaces input MEQdeacutepenses de fonctionnement nombre de megravetres carreacutes bruts input G10
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Environnement USNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
consommation de leacutenergie (exprimeacutee en BTU processus universiteacutes canadiennes et ameacutericaines BTUpopulation totale de luniversiteacute processus voir en particulier Penn State et Umich de leacutenergie produit par des sources renouvelable processusUtilisation de leau processuslitres deau par personne processusGaz agrave effet de serre (en tonnes meacutetriques CO2 eacutequivalents) processusMT CO2 par personne processusDeacutechets solides (en tonnes) processusTonnes de deacutechets solides per capita processus de deacutechets solides recycleacutes processus
RP gouvernance - 2008-06-23
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
10INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
ReacuteputationUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
enquecircte meneacutee aupregraves des eacutetudiants outcome NSSE Melbourneenquecircte meneacutee aupregraves des diplocircmeacutes outcomeenquecircte meneacutee aupregraves des pairs outcome USNews Asiaweek THESenquecircte meneacutee aupregraves des employeurs outcome Macleans Melbourneenquecircte meneacutee aupregraves des stakeholders outcome Macleans Melbourneenquecircte meneacutee aupregraves du monde des affaires outcome Macleans Melbournecouverture meacutediatique outcome AT
RP gouvernance - 2008-06-23
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
11INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Planification et gestion strateacutegiquesUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
Pouvoirs des corps universitaire processus un Canliens dynamiques entre Conseil Assembleacutee universitaire commission des eacutetudes processus un Canprocessus formel de planification strateacutegique processus un Canproduction de donneacutees et indicateurs processus un Canprocessus deacutevaluation des faculteacutes deacutepartements et services processus un Canrevue systeacutematique des responsabiliteacutes et efficaciteacute des comiteacutes processus un Candeacutepenses de ladministration centrale deacutepenses de fonctionnement processus un Can
RP gouvernance - 2008-06-23
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
12INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
INDICATEURS type universiteacutes
canadiennes et MEQ
classements nationaux classements internationaux
Eacuteducation permanenteUSNews and Report ampThe
CenterMacleans CHEDAAD GUARDIAN ASIAWEEK
DETYA AUSTRALIE
MELBOURNEREPUBLICA QS-TIMES Shanghai TAIWAN
WEB
nombre de programmes offerts input un Cannombre deacutetudiants inscrits input un CanRevenus et deacutepenses du service input un Can
RP gouvernance - 2008-06-23
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
13INVENTAIRE DES INDICATEURS COURANTSAgrave DES FINS DE PLANIFICATION DEacuteVALUATION DE LA PERFORMANCE ET DE REDDITION DE COMPTES
A = AlbertaO = OttawaQ = QueensT = TorontoU = UBCG10 = Consortium des Universiteacutes de recherche Canadiennes
REacuteFEacuteRENCES
Aperccedilu densemble des principaux indicateurs
GaitherG amp alii (1996) Measuring Up the Promises and Pitfalls of Performance IndicatorsAshe-Eric Higher Education Report No 5 Accegraves agravehttpericedgovERICDocsdataericdocs2sqlcontent_storage_010000019b8013f508pdfDavis D(1996) The Real World of Performance indicators A Review of Their Use in Selected Countries CHEMSUsher A amp Savino M (2006) A World of Difference Global Survey of University League tables EPI - Accegraves agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdfOffice of Planning amp Analysis Performance Indicators Discussion Paper McMaster University May 2004I (Indicateurs utiliseacutes dans les universiteacutes canadiennes Appendix B - Accegraves agrave httpwwwmcmastercapresrefiningperformance_indicatorspdfVan Dyke (2004) Indicators Used by Other PublicationsGovernment Departments to Rank or Rate Universities Melbourne InsituteAccegraves agrave httpwwwmelbourneinstitutecompublicationsreportsaus_uniAppendixBpdfPerformance Indicators in Select Higher Education Systems Appendix C 2000 Accegraves agrave httpwwwusgeduusg_statsbenchmarkscope1appendixCpdfMiddaugh M F National Study of Instructional Cost and Productivity The Delaware Study Accegraves agrave httpwww udel eduIRcostMiddaugh MF National Study of Instructional Cost and ProductivityThe Delaware Study Accegraves agrave httpwwwudeleduIRcost
Indicateurs utiliseacutes dans les classements nationaux ou par certaines universiteacutes
MEQ Contrats de performance Eacuteleacutements de suiviAsiaWeek ldquoAsiarsquos Best Universities Accegraves agrave httpwwwasiaweekcomasiaweekfeaturesuniversities2000The Guardian ldquoUniversity Guiderdquo Accegraves agrave httpeducationguardiancoukuniversityguide2008Macleans ldquoUniversity Rankingsrdquo Accegraves agrave httpwwwmacleanscauniversitiesUS News amp World ReportldquoAmericarsquos Best CollegesrdquoAccegraves agrave httpwwwusnewscomusnewseducollegerankingstankindex_briefphpTheCenterldquoThe Top American Research Universitiesrdquo Accegraves agrave httpthecenterufleduGovernment of Australia Characteristics and Performance Indicators of Higher Education Institutions Accegraves agrave httpwwwdestgovauarchivehigheredstatisticscharacteristicscontentshtmCHEDAAD University Ranking Accegraves agrave httpwwwdaaddedeutschlandhochschulenhochschulranking06543enhtmlmodule=BausteinClassement de La Republica httpwwwrepubblicaitspeciale2007guida_universitaindexhtmlEnvironmental Task force Report Proposed Environmental Performance indicators Michigan University Accegraves agrave http wwwumicheduprescomitteesetf_indicatorshtmlInfosource Accegraves agrave httpwwwresearchinfosourcecommedia2007RUY-AllUnivpdfUniversity of Toronto 2007 Performance Indicators for Governance Accegraves agrave httpwwwprovostutorontocapublicreportsperformanceindicators2007pihtmG10 Fact Book G10 Data Exchange (annuel)G10 Instructional and Financial Data at the Divisional Level Selected Depatments ARLAssociation of Research Libraries Accegraves agrave httpwwwarlorgbm~docarlstats06pdf
Indicateurs utiliseacutes dans les classements internationaux
Performance Ranking of Scientific Papers for World UniversitiesTaiwan Accegraves agrave httpwwwheeactedutwrankingindexhtmWEBOMETRICS Accegraves agrave httpwwwwebometricsinfomethodologyhtmlAcademic Ranking of World Universities Shanghai Jiao Tong University Institute of Higher Education Accegraves agrave httpedsjtueducnrankinghtmWorld University RankingsThe Times Higher Education Supplement (THES) Accegraves agrave httpwwwtopuniversitiescomworlduniversityrankingsuniversity_rankings_newsarticlethes_qs_world_university_rankings_methodologyLeiden Ranking Accegraves agrave httpwwwcwtsnlcwtsLeidenRankingWebSitehtml
RP gouvernance - 2008-06-23
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
ANNEXES
Quelques exemples de listes drsquoindicateurs
1 Universiteacute McMaster (2004) Performance Indicators Discussion Paper Appendix B List of indicators May 2004Acceacutedeacute agrave httpwwwmcmastercapresrefiningperformance_indicatorspdf (pp12-23) 2 Usher A amp Savino M(2006) A World of Difference A Global Survey of University League Tables Toronto Educational Policy Institute January 2006 Acceacutedeacute agrave httpwwweducationalpolicyorgpdfWorld-of-Difference-200602162pdf 3 Davis Dorothy (1996) The Real World of Performance Indicators A review of their use in selected Commonwealth countries London CHEMS March 1996 Appendix IV Core indicators recommended by the COU (Council of Ontario Universities)
Voir page suivante
Preacutepareacute par Roland Proulx
Consultant en planification institutionnelle et veille strateacutegique
Montreacuteal mai 2008
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
Performance Indicators
Discussion Paper
May 2004 (Prepared by the Office of Planning amp Analysis)
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
Table of Contents
Why Performance Measurement 2 Four Types of Indicators 4 Framework for Discussion 7 Appendix A Template 1 10 Template 2 11 Appendix B Table 1 ndash Undergraduate Education 12 Table 2 ndash Graduate Education 13 Table 3 ndash Research 14 Table 4 ndash Internal (Building Inclusive Internal Community) 15 Table 5 ndash Library 16 Table 6 ndash Information Technology (Information Infrastructure) 17 Table 7 ndash Internationalization 18 Table 8 ndash External (Linking with External Community Relationship amp Reputation Building) 19 Table 9 ndash Planning and Managing 20 Table 10 ndash Financial (Budget) 21 Table 11 ndash Faculty 22 Appendix C Guiding Principles 23 Notes 24 Reference Material 25
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
2
Table 1 ndash Number of Indictors by Type of Data Source
Table 1 shows that surveys are the least frequently used source of data for indicators
Indeed of all the studies only Hong Kongrsquos Education18 rankings come close to having a
plurality of indicators come from this source This measure somewhat underestimates the
importance of surveys however as it does not account for the weighting given to each
indicator in each study In the THES World Rankings for instance there is only a single
survey (for ldquoreputationrdquo) but it accounts for 40 of the total ranking Similarly Canadarsquos
Macleanrsquos rankings have only one survey-based indicator out of a total of 24 but this one
indicator is worth 20 of the final score
Raw
indIcator count
Surve
y data
Third parties
Universities
AsiaweekmdashAsias Best Universities 18 - - 18
Daily Telegraph (2003) 1 - 1 -
Education18com 9 3 4 2
Excelencia 2001 71 - 71 -
Financial Times (2003) 17 - 17 -
Guangdong Institute of Management Science 17 - 14 3
GuardianmdashUniversity Guide 2005 7 - 2 5
La Repubblica 23 2 21 -
Macleans University Rankings 24 1 5 18
Melbourne Institutemdash International Standing of Australian Universities
26 3 23 -
Netbig 2004 18 1 10 7
Perspektywy Rzeczpospolita Uniwersytet 18 1 2 15
Shanghai Jiao Tong UniversitymdashAcademic Ranking of World Universities
6 - 5 1
The TimesmdashGood University Guide 2005 9 - 9 -
Times Higher Education SupplementmdashWorld University Rankings
5 1 1 3
US News and World Reportmdash Americas Best Colleges 2006 15 1 3 11
Washington MonthlymdashCollege Rankings 2005 8 - 1 7
Wuhan University Centre for Science Evaluation 45 2 22 21
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
3
A Indicators of Beginning Characteristics
ldquoBeginning characteristicsrdquo refer to any part of the set of characteristics or abilities of
students at the time they begin their studies Fourteen of the 18 rankings examined in this
study use one or more indicators of the beginning characteristics of students to arrive at
their definition of ldquoqualityrdquo Of these the Washington Monthly puts the most emphasis on
these factors with 33 of the total ranking coming from this class of indicators but the
Guardian Education18 Asiaweek and the two other North American surveys also place
considerable emphasis on this category
There are six main indicators used to determine which institutions have students with
positive ldquobeginning characteristicsrdquo
The most common measure of beginning characteristics is performance on national
standardized tests with nine surveys using this as a measure Education18 and the
Guardian put the biggest emphasis on this measure (a weighting of 20) but it is also used
by the Melbourne Institute (11) Asiaweek (833) the US News and World Report (75)
Netbig (595) the Financial Times (5) the Times (33) and Wuhan (033) Because this
data is collected and standardized by national bodies it has the benefit of being seen as a
relatively impartial method of determining the relative ldquostrengthrdquo of the students entering
each institution Institutionsrsquo results can be scored by showing either averages or the
percentage of entering students meeting a particular standard
Canada is an exception to this rule as its main league table producermdashMacleanrsquosmdashuses
secondary school grades as a means of measuring the ldquostrengthrdquo of the student body This
is a second-best solution made necessary by the absence of any national standardized test
in Canada (or indeed of any provincial standardized tests at the end of secondary school
in provinces other than Alberta) The lack of national standardization makes this an
undoubtedly inferior indicator as there is no guarantee that an ldquoArdquo in one jurisdiction is
truly equivalent to an ldquoArdquo in another jurisdiction
Another measure of the strength of the student body is the percentage of incoming
students receiving (third-party) scholarships which is worth 11 of the score in the
Wuhan survey One can also approach the issue by measuring institutional selectivity In
effect this method infers the strength of the student body by the proportion of
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
4
applicants rejected the theory being that the higher the number of rejected applicants the
stronger the remaining students aremdashan approach forcefully critiqued by Peck (2003)
Normally this measure is expressed as a straight ratio of acceptances to applications but it
can also be expressed (as it is in Asiaweek which at 85 of the total puts by far the
greatest weight on this measure) as a ratio of enrolments to applications Within the US
there is some dispute as to what constitutes an offer of admission and whether or not late
admissions are included as noted by Ganeshananthan (2003)
Student bodies are often considered to be strong if the school is able to attract a large
number of international or out-of-district students or if they contain people from diverse
ethnic backgrounds A number of league tables use the international student indicator
(which like the selectivity indicator is arguably as much an indicator of prestige and
reputation as it is of student characteristics) although in no case does this indicator
account for more than 5 of the total ranking Only the Guardian uses ethnic diversity as a
quality indicator although othersmdashnotably the US News and World Reportmdashdisplay data
on this indicator without scoring it for inclusion in the final ranking At 8 the Guardian
puts a somewhat larger emphasis on this indicator in comparison to other league tables
which use similar variables
A very different take on this idea is present in the Washington Monthly which released its
first set of College Rankings in September 2005 With the declared aim of using an
institutionrsquos commitment to social mobility as a measure of quality it uses the percentage
of students from low-income backgrounds as an indicator (with percentage of students
receiving need-based government (Pell) grants used as a proxy)
Some measures of ldquobeginning characteristicsrdquo relate to the nature of studentsrsquo ldquostudy
statusrdquo Two of the Chinese rankings (Netbig and Wuhan) use an indicator based on the
percentage of the student population who are graduate students (arguably this is a
research ranking rather than a student one) In Polandrsquos Rzeczpospolita league table the
number of graduate students auditing classes is used as an indicator the assumption is
presumably that if people are auditing then the classes must be very attractive The Italian
La Repubblica ranks an institution according to the number of part-time students it has
contrary to prevailing North American views on the undesirability of part-time study the
Italian rankings see higher numbers of part-time students in a positive light as it is
evidence that an institution is becoming less rigid in its timetabling and
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
5
permitting students to juggle both work and study something which was nearly
impossible in that country just a few years ago
The Washington Monthly also has a final category of indicators which reflect studentsrsquo
beginning characteristics namely their likelihood of performing community service as
measured by the percentage of students in the US Peace Corps and Reserve Officer
Training Corps (ROTC) and the percentage of students involved in work-study in the
community3 Together these three indicators account for 33 of an institutionrsquos total
ranking
B Indicators of Learning InputsmdashStaff
Generally both the quantity and quality of staff are positively correlated with institutional
quality The problem of course is finding useful metrics for each of these factors
especially if one excludes as we have done here measures of research performance and
research intensity putting them in a separate category4
The simplest measure is simply the number of faculty unadjusted for things like size of
student body Most national league tables however prefer to use variations on the
concept of facultystudent ratio Others try to measure teaching intensity with measures
such as courses per teacher or hours spent in class per student (both in La Repubblica)
These kinds of measures usually account for between 2-5 of the final rankings although
in some cases (ie the Guardian) this figure can be as high as 20
Another important way of measuring how faculty resources are deployed is the measure
of average class size which is used only by Macleanrsquos and the US News and World Report
Ostensibly the reason for measuring class size is to account in some form for the degree
3 Judging by the text that accompanies its rankings the authors of the Washington Monthly rankings would probably disagree with the classification of these measures as ldquobeginning characteristicsrdquo since they clearly intend them to be a measure of the institutionrsquos commitment to community service rather the students Our judgement however is that in the end the decision to join the Peace Corps or the ROTC rests with the individual student and the institution so far as we can tell does not play a significant role in the enrolment process Similarly although institutions are responsible for allocating work-study money it is generally speaking up to the student who qualifies for work-study to find or create a job on his or her own whether in the community or on campus On balance we feel that these indicators can more accurately be said to reflect the inclinations and decisions of the students rather than those of institutions and hence belong in the ldquobeginning characteristicrdquo category rather than the ldquolearning inputsmdashresourcesrdquo category 4 Indeed the dividing line between ldquoLearning InputsmdashStaffrdquo and ldquoResearchrdquo is a difficult one to enforce especially with respect to indicators which attempt to look at the quality of staff by measuring research Our litmus test is as follows if the indicator refers to a professorrsquos accomplishments as a researcher (eg membership in an academy some kind of third-party research award) we have included it in the research category rather than the staff category
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
6
of attention devoted to actually teaching students Despite the fact that important research
(Gilbert 1995) has cast doubt on class size as a proxy for quality at the institutional level
the use of this indicator appears to be a spillover from the North American debates on
class sizes at the primary and secondary levels (see Krueger Hanushek and Rothstein
2000) Regardless of why the indicators are used they are extraordinarily important to
these two rankings systems making up 14 and 8 of the Macleanrsquos and US News and
World Reportrsquos rankings respectively
A number of ranking systems try to look at staff qualifications such as the number of
PhDs or tenure-track staff employed (Asiaweek Netbig Education18 Macleanrsquos the
Washington Monthly5 and the US News and World Report) Macleanrsquos goes one step further
than other surveys and actually looks at the proportion of classes taught by tenure-track
staff Others (ie THES) look at the number of foreign faculty based on the assumption
that institutions with higher numbers of foreign staff must be ldquoattracting qualityrdquo Still
others (ie La Repubblica) look at the age structure of the faculty Another proxy for
institutional quality is the pay rates for tenured staff on the assumption that institutions
with higher rates of pay on average attract better faculty this measure has been used both
by the US News and World Report and Asiaweek
Finally a number of league tables rank faculty inputs on the basis of standardized third-
party evaluations 6
Education18 the Financial Times the Times the Guardian and the US
News and World Report league tables all use some sort of ranking criterion based at least in
part on this indicator or variations thereof
C Indicators of Learning InputsmdashResources
Resource inputsmdashcrudely the amount of current dollars equipment and books available
to students at an institutionmdashare widely considered an important measure of quality Yet
despite the apparent simplicity of counting dollars and measuring assets
5 Only hard sciences and engineering PhDs are considered No participation from any other subject area counts 6 Until 1997 the Quality Assessment Agency provided regular Teaching Quality Assessments of each department of each
university Since that date the TQA has not been updated in a consistent way (participation was in effect made voluntary in 1997) Since a number of UK league-table producers relied on this data the end of the TQA led to a reduction in the number of media organizations releasing league tables from four papers only a few years ago down to the current two (the Guardian and the Times) Neither the Daily Telegraph nor the Financial Times have issued university league tables at all in the last two years and there is no indication that either will be updated in the future
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
7
the means by which institutional wealth is measured varies considerably between ranking
systems
There are a number of revenue-based measures of resources Macleanrsquos uses public funding
of institutional budgets as a factor in its analysis conversely the Financial Times uses the
private funding of institutional budgets as an indicator of quality Both Macleanrsquos (3 of
total score) and the US News and World Report (5 of total score) also measure alumni
financial support as a measure of quality
For reasons that are not entirely clear league tables tend to favour measures of
expenditures rather than revenues The Guardian looks at total institutional expenditures
as an indicator Institutional expenditure on student services is used as a measure of
institutional quality by both the Times and Macleanrsquos (counting for 33 and 43 of total
institutional scores respectively) Rzezspospolita does not measure student services
expenditures directly but does measure student services outputs such as number of
student athletes and number of study clubs which amounts to more or less the same thing
Macleanrsquos also gives out 433 of its total score based on institutional expenditures on
scholarships and bursaries
Various aspects of physical infrastructure are also used as measures of institutional
resources most directly in the case of La Repubblica which bases 317 of its total rank on
the number of lecture spaces at an institution Rather cryptic measures of ldquobuilding
assetsrdquo are also used by two Chinese ranking systems (Netbig and Wuhan) Another type
of physical infrastructure measured is available Internet bandwidth which was used by
Asiaweek in its now-defunct rankings Generally speaking all of these measures are worth
roughly 3 of the total score
By some distance the infrastructure indicators most favoured by the compilers of league
tables are library resources The Macleanrsquos rankings put perhaps the most emphasis on
this with 12 of the total quality mark being taken from various types of library
infrastructure measurements (including acquisitions per year total volumes average
number of volumes per student and yearly library expenditure outside of acquisitions)
Netbig and Education18 also use library volume holdings while Asiaweek the Financial
Times and the Times also use measures of library expenditures outside of acquisitions or
computerization of library resources as measures of institutional quality
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
8
One important factor to note is that most ranking systems do not normalize their resource
and infrastructure measures That is to say it is raw spending power or simple size of
assets that is usually measured rather than spending per studentprofessor or assets per
studentprofessor As a result a number of these rankings systems have inbuilt biases
towards larger institutions
D Indicators of Learning Outputs
Learning outputsmdashthat is measurements of educational attainment or of skillsknowledge
learned over the course of a baccalaureate degreemdashshould be a basic indicator of
institutional quality Unfortunately good means of measuring these outputsmdashlike the
National Survey of Student Engagement (NSSE) and the College Learning Assessment
(CLA)mdashhave only recently become available and for the most part institutions are still
keeping their scores secret Outside of these measures only a few very crude indicators are
available which likely explains why learning outputs do not feature especially prominently
in most ranking schemes
The simplest types of measures of learning outputs are those linked to graduation and
retention rates The US News and World Report La Repubblica Macleanrsquos Wuhan Guangdong
and the Melbourne Institute all use undergraduate graduation rates as proxies for quality7
the latter three also use rates of graduation from Masterrsquos programs as indicators In some
cases the weights on these measures can be very highmdashin the Guangdong rankings
graduation rates account for over 50 of the rankingmdashbut in most cases the weights are
10 or less Retention rates commonly meaning the progression rate of first-year students
into second year are accorded less importance The US News and World Report Macleanrsquos
the Melbourne Institute and La Repubblica all employ retention measures as indicators but
none of them are worth more than 4 of total weighting Two publications make specific
indicators for retention and graduation of international students Macleanrsquos (graduation
rates of international students) and the Melbourne Institute (retention rates of international
students) The Washington Monthly looks specifically at institutional retention rates
adjusted for the participation of lower-income students and gives higher scores to
institutions whose rates significantly exceed their ldquopredictedrdquo values based on SAT scores
and number of Pell Grant recipients the US
7 Usually the time-to-graduation is time-delimited so only those students who graduate in under for example six years are counted on these measures The Washington Monthlyrsquos measure is designed to serve a slightly different purpose and based on another metric for academic performance using changing graduation rates over time Please see the section on Beginning Characteristics on p 18
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
9
News and World Reportrsquos graduation rate performance indicator and the Guardianrsquos
ldquovalue-addedrdquo indicator also score institutions on a real vs predicted basis)
E Indicators of Final Outcomes
Final outcomes are indications of generalized outcomes for students after graduation
Finnie and Usher (2005) state that these outcomes are in theory unlimited (eg happiness
good citizenship) but given the somewhat utilitarian justifications for education that are
currently in fashion (see Wolf 2000) employment outcomes are the most commonly used
measure of final outcomes These are given particular emphasis by the Guardian (where
employment outcomes are worth 17 of the total score) but are also used by the Financial
Times (6) the Times (33) and Wuhan (06) The Guardian the Financial Times and the
Times are interestingly not concerned with employment per se but with ldquoemployment in
an area relevant to onersquos course of studiesrdquo The Guardian using data from the Higher
Education Statistics Agency (HESA) uses Standard Occupational Classifications to
measure the proportion of graduates in professional or white-collar jobs anyone not in
such a job is considered not to be working in an area related to their studies (it is unclear
what methodology is used by the Financial Times and the Times although we suspect their
methods are broadly similar)
The only other measure of final outcomes in use is percentage of graduates returning for
additional education which is an indicator used by both the Melbourne Institute and the
Financial Times This is a particularly important indicator for the latter as it is worth 21
of the final ranking
The lack of indicators concerning final outcomes is interesting since most government-
sponsored performance-indicator regimes around the world are very much concerned
with such measures especially with respect to employment Possibly this indicates that
ranking systems simply do not view education outcomes as relevant measures of
educational quality Alternatively it may be the case that they simply have not found a
reliable indicator of outcomes or that there are reliable indicators but that there is so little
variation between institutions that it makes no sense to rank based on the data
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
10
F Indicators of Research
Many of the league tables covered in this survey include sections and weightings related
to universitiesrsquo research efforts It is in this field of measurement that we see the greatest
diversity of indicators Presumably this is because research inputs and outputs lend
themselves much more easily to measurement and manipulation than other areas of
institutional activity
Three studies include research staff as part of their ranking scheme La Repubblica at 952
the Melbourne Institute (4) and Wuhan (078)
Bibliometricsmdashthat is the counting of publications and citationsmdashis one commonly used
method of looking at research quality but it is not universally admired because different
disciplines use different means to communicate major advances in knowledge (leading
scientists invariably produce large numbers of journal articles leading social scientists
may produce fewer journal articles but instead have one or two long important
monographsmdashsee Hicks 2004) There is also some concern among non-English speaking
countries that they are penalized in international rankings because so many of the major
journals (notably Science and Nature) are printed in English However the one set of
rankings that uses separate indicators to monitor articles published in English and articles
published in another language (the Wuhan rankings) shows that the two indicators are
positively correlated institutions that have more Chinese publications are also likely to
have more English publications and vice versa
Several sets of league tables measure bibliometric citations in various publication indices
The Shanghai Jiao Tong and the THES rankings both emphasize this category by giving it a
weight of 20 of the final total Guangdong also monitors other Chinese universities
specifically for citations in engineering publications and weights this at 29 Moreover it
tacks on an additional 1049 for citations in science-oriented indices such as the Science
Citation Index The Shanghai Jiao Tong rankings are close behind Guangdong at 10 for the
same category of scientific citations while the Melbourne Institute rates science citations at
68 and Wuhan at 128 Citations in social scienceoriented indices (ie the Social
Science Citation Index which does not include the humanities) are noted in only two
league tables those of Shanghai Jiao Tong (10 of the final weighting) and the Melbourne
Institute (32) Another way of measuring research
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
11
impact is to focus specifically on citations in ldquohighly citedrdquo publications8 These are given a
weighting of 20 by the Shanghai Jiao Tong rankings 54 by Wuhan and 2 by the
Melbourne Institute
The complement of citations is of course publications Listing the number of publications
an individual group department or whole university releases can act as a weak substitute
for citationsmdashweak because simply publishing a paper or monograph is no guarantee that
the general public or other researchers will even glance at the work Guangdong gives an
1179 weighting to publications in science-oriented indices such as the Science Citation
Index from Thomson-ISI Similarly 136 of Netbigrsquos ranking is based on the same
indicator while the Melbourne Institute weights this at 4 and Wuhan at 146 Guangdong
even has a separate category just for measuring publications in Science and Nature
although it accords it an almost derisory weighting of 06 Under publications in social
science-oriented indices Netbig adds another 84 and the Melbourne Institute 2 to their
final totals For publications in other indices (where the subject indices are
undifferentiated) the weighting is 66 for Asiaweek 5 for Education18 45 for
Guangdong and 145 for Wuhan As for other publications Asiaweek was the only set of
rankings to include research monographs weighted at 033
In countries where there are specific third-party evaluations of research output academic
quality of research is sometimes used as a research indicator The Times puts a very large
30 weight on this indicator while the Financial Times puts it at 119
Research awards are another handy third-party measurement of quality as the number of
international and national awards won by faculty andor graduates is often considered a
useful measure of institutional success International research awardsmdash specifically the
number of alumni who have won Nobel Prizes or Fields Medalsmdashare used as an indicator
by Shanghai Jiao Tong and at 30 given enormous weight This indicator is seen as
particularly suspect in some quarters given that the points are based on where the
recipient went to school rather than on where they are or were on the
8 The definition of ldquohighly citedrdquo has been standardized for the purposes of comparison by Thomson-ISI suppliers of the
most prominent publication indices
9 The two English guides use the 2001 Research Assessment Exercise (RAE) results from Britainrsquos funding councils which rank each university using a graduated scale from 1 (bottom) to 5 (top) Melbournersquos International Standing paper judges academic research quality through the use of the Essential Science Index for both the hard and soft sciences
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
12
faculty and that some of the Nobellists propping up institutionsrsquo rankings have been dead
for nearly a century Wuhan uses similar measures but only accords them a weight of 14
National research awards are more common as a quality indicator used by La Repubblica
(952) the Melbourne Institute (8) Wuhan (713) Netbig (4) and Guangdong (156)
Financial indicators of research are also very common Research budgets as a factor in the
overall assessment of research in universities are covered by the Financial Times (9)
Netbig (6) and the Melbourne Institute (333) Wuhan lists a figure of 178 allocated for
total amount of research expenditure unfortunately it is unclear precisely what this
research expenditure represents or how it is determined although it is clearly indicated
that it does not represent the total number of grants or projects at a university Total
number of research-based grants and projects is weighted by Education18 at 15 and
Wuhan at 931 Macleanrsquos devotes 55 of its weight to public-source grants for science
and engineering and another 55 to those for social sciences and humanities Similarly
the Melbourne Institute gives 6 of its overall weight to public-source grants making no
distinction between areas of study
In a slightly different vein Netbig (46) and Wuhan (278) both list the number of
research-based chairs per institution Also Netbig (86) Wuhan (548) and La Repubblica
(095) all weigh research-basedaffiliated research institutions or centres for studies
Finally one can also measure research not simply in terms of the amount of money it
generates but also in terms of the amount of future income it will generate Both
Guangdong (245) and Wuhan (193) measure the number of patents issued to
universities as a quality indicator
A final way of measuring an institutionrsquos research intensity is to look at the range of its
course offerings Asiaweek (3) Netbig (68) and Wuhan (195) all use the number of
doctoral and Masterrsquos programs offered as a proxy for research intensity
As with physical and financial resources few if any of the research indicators are
normalized to account for institutional size (either by student or faculty numbers) In the
world of rankings bigger almost always means better an institution with 100 faculty with
ten citations apiece will always look worse than an institution with 1001 faculty
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
13
with one citation each To the extent that the raw production of knowledge matters this
form of measurement is acceptable To the extent that rankings are meant to show how
well institutions are doing on a like-to-like basis or to show the efficiency of universities it
is plainly inadequate This should be of particular concern to Chinese policy-makers
whose ranking systems are especially reliant on research-based indicators
G Indicators of Reputation
The final set of indicators for quality ranking schemes is ldquoreputation and peer appraisalrdquo
Those rankings systems which use the results of reputation surveys as an indicator do so as
an indirect measure of quality based on the assumption that the employers academics and
academic administrators surveyed have opinions of institutional quality that are informed
up-to-date and impartial While these assumptions are clearly open to debate they
nevertheless form an important basis for many ranking systems Another reason for using
reputation measures is the paucity of other data availablemdashsome countries have few
independent measures of teaching effectiveness university resources or output and
reputation can thus act as a useful surrogate Reputation rankings are often criticized as
simply quantifying the common ignorance of the people being surveyed However to the
extent that the people being surveyed hold positions which have the potential to affect
large numbers of young people and whose positions actually require some knowledge of
institutional quality (ie officials in charge of graduate admissions corporate recruiters
etc) then reputation rankings make sense because they provide useful information for
students about the perceived value of the degrees that they could obtain from various
universities
The greatest emphasis on reputation is found in the rankings of Perspektywy in Poland and
the Times which both accord reputation a weighting of 50 in their overall ranking
scheme Education18 assigns it almost as much significance at 40 The US News and World
Report applies a weight of 25 followed closely by Asiaweek at 20 Clustering tightly just
below these league tables are the trio of the Melbourne Institute (171) Macleanrsquos (16)
and Netbig (15) The only other study to include reputation is Wuhan (117)
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
14
1 Please see Appendix B for a glossary of the various bodies referenced in this document
Appendix A Detailed Listing of Indicators and their Sources
Beginning Characteristics Indicator
Used By Source1
Incoming grades Macleans University
Percentage with grades above a set limit
Macleans University
US News and World Report
University
Performance on national standardised tests or benchmarks
Asiaweek University
Education18 3rd-party JUPAS
Financial Times Government agency 3rd-party UCAS
Guardian University Guide
Government agency 3rd-party UCAS
Melbourne Institute Government agency 3rd-party DEST
Netbig National entrance examination board
Times Good University Guide
Government agency 3rd-party UCAS
US News University
Wuhan Unknown presumed government 3rd-party
Student status La Repubblica Government agency 3rd-party MIUR
Admittance selectivity general
Asiaweek University
Admittance number of applications to places
Asiaweek University
Financial Times Government agency 3rd-party UCAS
La Repubblica Government agency 3rd-party MIUR
US News University
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
15
Beginning Characteristics Indicator
Used By Source1
Out-of-locality student percentage
Macleans University
International student percentages
Financial Times Government agency 3rd-party HESA
Macleans University
Shanghai Institute of Educational Science
Times World University
Wuhan Unknown presumed university
Undergraduate students among all students percentages
Netbig
Wuhan Unknown presumed university
Ethnic diversity in student body
Guardian University
Learning Inputs ndash Staff Indicator
Used By Source
Facultystudent ratio Asiaweek University
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Financial Times Government agency 3rd-party HESA
La Repubblica Government agency 3rd-party MIUR
Times Good University Guide
Government agency 3rd-party HESA
Times World University
US News University
Wuhan Unknown presumed university
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
16
Learning Inputs ndash Staff Indicator
Used By Source
Social science faculty student ratio
Melbourne Government agency 3rd-party DEST
Science faculty student ratio
Melbourne Government agency 3rd-party DEST
Administrative staff student ratio
Excelencia Government agency 3rd-party Centro de Investigaciones Socioloacutegicas
Staff student ratio (regardless of division)
Guardian Government agency 3rd-party HESA
Netbig University
Course per teacher La Repubblica Government agency 3rd-party MIUR
Per-teacher university spending
Asiaweek University
Faculty pay rates fortenured staff
Asiaweek University
US News University
Number of full-time part-time faculty
Netbig University
US News University
Wuhan Unknown presumed university
Faculty with researchprojects
Wuhan Unknown presumed university
Class size differentiation Macleans University
US News University
Classes taught by tenured faculty
Macleans University
Exchange programmes hosted
La Repubblica Government agency 3rd-party AgNaSoc
Number of classes lsquoactually taughtrsquo
La Repubblica Government agency 3rd-party MIUR
of international faculty (v faculty as a whole)
Times World University
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
17
Learning Inputs ndash Staff Indicator
Used By Source
Aging and staffreplacement churn issues
La Repubblica Government agency 3rd-party MIUR
Teaching quality Faculty performance on standardised 3rd-party tests if given
Education18 3rd-party TLQPR
Financial Times Government agency 3rd-party QAA HESA
Times Good University Guide
Government agency 3rd-party QAA HESA
US News University
Teaching quality Performance on own metrics
Guardian Survey (cobbled together from QAA scores)
Teaching quality Qualifications for teaching positions (PhDs Masters etc)
Asiaweek University
Education18 University
Macleans University
Netbig University
US News University
Number of doctoral and Masters programmes
Asiaweek University
Netbig Unknown presumed university
Wuhan Unknown presumed university
Student efforts Hours spent in class per student
La Repubblica Government agency 3rd-party CNVSU
Student efforts student participation in exchange projects
La Repubblica Government agency 3rd-party AgNaSoc
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
18
Learning Inputs ndash Resources Indicator
Used By Source
Physical infrastructure Number of lecture spaces
La Repubblica Government agency 3rd-party MIUR
Physical infrastructure Library Acquisitions per year
Macleans University
Physical infrastructure Library total volumes
Education18 University
Macleans University
Netbig Unknown presumed university
Physical infrastructure Library volumes per student
Macleans University
Physical infrastructure Library Yearly expenditures outside of acquisitions
Asiaweek University
Financial Times Government agency 3rd-party HESA
Macleans University
Physical infrastructure Internet bandwidth
Asiaweek University
Physical infrastructure Computerisation of library resources
Asiaweek University
Financial Times Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Public funding total of institutional budget
Macleans University
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
19
Learning Inputs ndash Resources Indicator
Used By Source
Funding and financial resources Private funding total (including supporting foundations and charitable organisations)
Financial Times Government agency 3rd-party HESA
Funding and financial resources Alumni support
Macleans University
US News University
Funding and financial resources Student services
Macleans University
Times Good University Guide Government agency 3rd-party HESA
Funding and financial resources Science grants
Macleans University
Funding and financial resources Social sciences and humanities grants
Macleans University
Funding and financial resources Expenditure
Guardian Government agency 3rd-party HESA
Shanghai Institute of Educational Science
Funding and financial resources Bursaries and scholarships disbursed by public private bodies
La Repubblica Government agency 3rd-party MIUR
Macleans University
Shanghai Institute of Educational Science
Wuhan Unknown presumed university or government agency 3rdparty
Funding and financial resources
La Repubblica Government agency 3rd-party AgNaSoc
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
20
Learning Inputs ndash Resources Indicator
Used By Source
Awards (not research awards) subsidised or unsubsidised
Macleans University
Learning Outputs Indicator
Used By Source
Academic performance Guardian Government agency 3rd-party plus university (so-called value-added measure)
Shanghai Jiao Tong University
Times Good University Guide Government agency 3rd-party HESA
US News University
Graduation rate Undergraduates only
Guangdong Institute of Management Science
Unknown
La Repubblica
Macleans University
Melbourne Institute Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Masters only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
Graduation rate Doctoral students only
Guangdong Unknown
Melbourne Government agency 3rd-party DEST
Wuhan Unknown presumed university
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
21
Learning Inputs ndash Resources Indicator
Used By Source
Graduation rate International students
Macleans University
Type of degree obtained Financial Times Government agency 3rd-party HESA
Retention 1st to 2nd year
La Repubblica Government agency 3rd-party CNVSU
Macleans University
Melbourne Institute Government agency 3rd-party DEST
US News University
Final Outcomes Indicator Used By Source
Work status Financial Times Government agency 3rd-party HESA
Guardian Government agency 3rd-party HESA
Times Good University Guide Government agency 3rd-party HESA
Wuhan Unknown presumed survey or government agency 3rd-party
Further professional education
Financial Times Government agency 3rd-party HESA
Melbourne Survey government agency 3rd-party DEST
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
22
Research Indicator Used By Source
Research staff numbers or percentage of research personnel (ie as opposed to teaching staff)
La Repubblica
Melbourne Government agency 3rd-party - there is some suggestion on researchers part that this data is obsolete DEST
Wuhan Unknown presumed government agency 3rd-party
Academic quality of research
CUAA Unknown
Financial Times HEFC Northern Ireland Higher Education Council (NIHEC) SHEFC
Melbourne 3rd-party DEST ESI (lab amp non-lab) University -administered survey of postgraduates
Times Good University Guide
Awards International Shanghai Jiao Tong
Wuhan Unknown presumed government agency 3rd-party
Awards National Guangdong Unknown presumed government agency 3rd-party
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed government agency 3rd-party
Awards Regional (ie stateprovincial or within national borders)
Guangdong Unknown presumed government agency 3rd-party
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
23
Research Indicator Used By Source
Citations Science-oriented indices (ie the Science Citation Index refers to natural sciences engineering and other related fields)
Guangdong Unknown presumed government agency 3rd-party CSCD (China) SCI Nature Science
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party SCI CSTPC
Citations Social science-oriented indices (ie the Social Science Citation Index and not the humanities) ndash
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong 3rd-party
Citations Highly cited (as determined by Thomson-ISI)
Melbourne 3rd-party Non-lab ESI
Shanghai Jiao Tong
Wuhan 3rd-party ISI-related indices
Citations Other Asiaweek 3rd-party
Shanghai Jiao Tong 3rd-party
Times World 3rd-party
Wuhan 3rd-party CSTPC CSSCI SCI SSCI amp AHCI
Publications Nature and Science (not quite the same as lsquohighlycitedrsquo above)
Guangdong Unknown presumed government agency 3rd-party ndash Nature and Science
Publications Published papers in science-oriented indices (ie the Science Citation Index)
Guangdong Unknown
Melbourne 3rd-party Lab ESI
Netbig 3rd-party SCI Engineering Index
Wuhan 3rd-party CSTPC SCI
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
24
Research Indicator
Used By
Source
Publications Published in social science-oriented indices (ie the Social Science Citation Index)
M elbourne
3rd-party Non-lab ESI
Netbig
3rd-party SSCI
Publications Published papers in other indices ndash
Asiaweek
3rd-party
Education18
3rd-party RGC
Guangdong
Unknown
Wuhan 3rd-party AHCI and others not described fully
Publications Books (other)
Asiaweek 3rd-party
Research budget including grants
Asiaweek University
Financial Times
Government agency 3rd-party RAE 2001
Research budget Expenditure (undefined)
Wuhan
Unknown presumed survey or university
Research budget Total number of grants and projects
Education18
3rd-party RGC
Wuhan
Government agency 3rd-party NSF(c) and NSSF(c)
Patents
Guangdong
Unknown
Wuhan
Unknown presumed government agency 3rd-party
Number of research-based chairs per institution
Netbig
Government agency 3rd-party
Wuhan
Unknown presumed government agency 3rd-party
Number of research-basedaffiliated research institutions centres for studies etc
La Repubblica
Netbig Government agency 3rd-party
Wuhan Unknown presumed university
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
25
Research Indicator
Used By
Source
Other output
Guangdong
Unknown
Wuhan
Unknown
Reputation Indicator Used By Source
Among studentsgraduates
Melbourne Survey
Among academics Asiaweek Survey
Education18 Survey
Netbig Survey
Times World Survey
US News Survey
Wuhan Survey
Among general society business sector others outside direct connection to university
Education18 Survey
Macleans Survey
Melbourne Survey
Wuhan Survey
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
The Real World of PERFORMANCE INDICATORS
A review of their use in selected Commonwealth countries
Prepared by Dorothy Davis Director International Students Division
IDP Education Australia
For CHEMS Commonwealth Higher Education
Management Service 36 Gordon Square
London WC 1H0PF
March 1996
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
DEUXIEgraveME PARTIE
CHOIX DrsquoINDICATEURS DE PERFORMANCE INSTITUTIONNELS
RELATIF Agrave UN EXERCICE DE REDDITION DE COMPTES
utilisation des ressources
qualiteacute de lenseignement
activiteacutes de la recherche
PREPARE PAR ROLAND PROULX
CONSULTANT EN PLANIFICATION INSTITUTIONNELLE ET VEILLE STRATEGIQUE
MONTREAL MAI 2008
wwwigopporg 2
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
wwwigopporg 3
Le choix drsquoindicateurs de performance institutionnels relatifs agrave un exercice de reddition de
comptes dans les universiteacutes est soumis agrave un certain nombre drsquoexigences et confronteacute agrave de
nombreux deacutefis parmi lesquels figurent les suivants
Les indicateurs de performance doivent ecirctre choisis et produits en eacutetroite relation avec un processus formel de planification strateacutegique ougrave notamment les eacutenonceacutes de mission et de vision ainsi que lrsquoeacutetablissement des objectifs et des cibles strateacutegiques constituent la raison drsquoecirctre la mise en perspective et le cadre opeacuterationnel
Le nombre des critegraveres que retient lrsquoexercice proposeacute de reddition de compte a eacuteteacute limiteacute agrave trois (3) ceux‐ci refleacutetant les pratiques courantes
Les nombreux exercices de reddition de comptes reacutealiseacutes aussi bien dans les universiteacutes que dans le monde des affaires font eacutetat de lrsquoimportance de limiter le nombre des domaines drsquoapplication des critegraveres et des indicateurs leur nombre varie entre six (6) et vingt (20)
Pour eacuteviter les confusions et les ambiguiumlteacutes des deacutefinitions claires preacutecises et opeacuterationnelles doivent ecirctre eacutetablies et refleacuteter ce qui est reconnu au sein notamment des Bureaux de recherche institutionnel et des Banques de donneacutees gouvernementales et priveacutees au niveau national et international
Le choix des indicateurs doit obeacuteir agrave un eacutequilibre parmi les types drsquoindicateurs (input processus output et outcome) Les indicateurs drsquoinput et de processus sont drsquoabord des preacutedicteurs de reacuteussite et non des reacuteussites elles‐mecircmes les indicateurs de reacutealisation et de reacutesultats devront donc avoir une preacutefeacuterence dans un exercice de reddition de comptes faisant surtout eacutetat de laquo performance raquo et de productiviteacute
La production des indicateurs doivent pouvoir compter sur des donneacutees facilement et commodeacutement accessibles
Enfin les indicateurs de performance doivent pouvoir ecirctre compareacutes agrave lrsquoeacutechelle tant reacutegionale et nationale qursquointernationale La mesure de la performance doit certes ecirctre eacutetablie par rapport agrave la reacutealisation des objectifs strateacutegiques drsquoune institution mais doit eacutegalement pouvoir se reacutefeacuterer aux meilleures pratiques tant nationales qursquointernationales des institutions comparables
Le choix des indicateurs de performance qui est proposeacute a voulu tenir compte du deacutefi de
reacuteunir des indicateurs refleacutetant ces exigences
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
1
Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Utilisation des ressources (16 indicateurs)
des professeurs et chercheurs avec PhD (ou grade terminal) input universiteacute SYSPER nationalnombre deacutetudiants EETC Personnel enseignant eacutequivalent temps complet input universiteacute nationaleacutetudiantspersonnel de soutien (selon les diverses cateacutegories de personnel) input universiteacute national
subvMEQrevenus de fonctionnement input universiteacute SIFU national droits de scolariteacute revenus de fonctionnement input universiteacute national revenus de fonctionnement eacutetudiant en eacutequivalence temps complet input universiteacute national fonds de dotation eacutetudiants en eacutequivalence temps complet input universiteacute national
salaire moyen des professeurs input universiteacute SYSPER nationalsalaire moyen des personnels (staff) selon les diverses cateacutegories input universiteacute SYSPER national
Nombre de megravetres carreacutes nets megravetres carreacutes nets alloueacutes input universiteacute MEQ SILUX nationalMCB eacutetudiants en eacutequivalence temps complet input universiteacute MEQ SILUX national
Deacutepenses totales consacreacutees aux TIC input universiteacute national
Deacutepenses des bibliothegravequesbudget de fonctionnement input universiteacute Nord-ameacutericain
budget alloueacute agrave lenvironnement input agrave construirehellip plusieurs universiteacutes nord-ameacutericaines mesures environnementales BTU litres deau MT CO2 deacutechets solides per capita processus agrave construirehellip
eacutequilibre budgeacutetaire chargesproduits processus universiteacute SIFU national
RP gouvernance-reddition de comptes2008-07-08
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
2Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Qualiteacute de lenseignement (29 indicateurs)
Nombre deacutetudiants en nombre absolu et en eacutequivalence temps complet input universiteacute RECU international
dossier scolaire des nouveaux eacutetudiants agrave lentreacutee (bulletin Cote R) input universiteacute MEQ national inscriptionsdemandes dadmission input universiteacute international
des eacutetudiants internationaux de 1er cycle et de 2-3e cycles input universiteacute international eacutetudiants reacutesidents du Queacutebec agrave leacutetranger input universiteacute MEQ
perseacuteveacuterants en 2e anneacutee du baccalaureacuteat output universiteacute national diplomationbaccalaureacuteat apregraves 4 ou 5 ans (selon le programme) output universiteacute G10 nationalinternational dilplomationmaicirctrise apregraves 4 ans output universiteacute G10 national diplomationdoctorat apregraves 6 ans output universiteacute G10 national
Deacutepenses de fonctionnement par eacutetudiant en ETC input universiteacute nationalinternationalnombre deacutetudiants EETC Professeur reacutegulier plein temps input universiteacute nationalinternational deacutepenses denseignement deacutepenses totales de fonctionnement input universiteacute nationalinternational
Nombre dordinateurs par eacutetudiant output universiteacuteNombre de cours en ligne processus universiteacutenombre de salles de cours eacutequipeacutes (audiovisuel postes en accegraves libre) output universiteacuteclassement international du site WEB outcome WEBOMETRIC international
nombre total de volumes et documents audio et eacutelectronique eacutetudiant en ETC input universiteacute ARL nord-ameacutericainclassement nord-ameacutericain des bibliothegraveques outcome universiteacute ARL ARL nord-ameacutericain
nombre de cours dispenseacutes par lInstitution par eacutetudiant input universiteacute cours assumeacutes par les professeurs reacuteguliers input universiteacute Macleans nationalTaille des groupe cours input universiteacute Macleans national
Education permanente nombre de programmes offerts input universiteacute
taux demployabiliteacute apregraves 1 an outcome universiteacute MEQ universiteacutes ontariennesValeur ajouteacutee comparaison du diplocircme avec le dossier agrave lentreacutee outcome
Eacutevaluation des uniteacutes denseignement et de recherche par les pairs processus universiteacuteEacutevaluation des services processus universiteacuteenquecircte meneacutee aupregraves des eacutetudiants outcome universiteacute NSSE nord-ameacutericainenquecircte meneacutee aupregraves des pairs outcome universiteacute Macleans THES internationalenquecircte meneacutee aupregraves des employeurs outcome universiteacute Macleans THES international
RP gouvernance-reddition de comptes2008-07-08
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-
3Choix dindicateurs de performance institutionnels relatif agrave un exercice de reddition de comptes
trois critegraveres utilisation des ressources qualiteacute de lenseignement et activiteacutes de la recherche
INDICATEURS TYPE SOURCE REPEacuteRAGE
INTERNE EXTERNE
Activiteacute de recherche (13 indicateurs) universiteacute SIRU amp alii
de professeurs et chercheurs ayant obtenu une subvention input universiteacute G10 nationalRevenus de recherche subventionneacutee professeurs reacuteguliers et chercheurs plein temps input universiteacute STAT CANCREPUQInfosource national
Nombre de publications par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par professeur et chercheur outpput universiteacute OST amp ISI Thomson Scientific nationalinternationalNombre de citations par publication outpput universiteacute OST amp ISI Thomson Scientific nationalinternational
Prix et meacutedailles de niveau national et international outpput universiteacute G10 amp IMU amp Macleans nationalinternationalNombre de PhD et de maicirctrise de recherche deacutecerneacute par professeur outpput universiteacute G10 nationalnombre de brevets et de licences outpput universiteacute G10 nationalnombre de spin-off companies outpput universiteacute G10 national
Nombre de chaires et de centres dexcellence outpput universiteacute gouv feacuted Programme de chaires nationalFondation canadienne pour linnovation (FCI) outpput universiteacute gouvernement feacutedeacuteral (FCI) national
nombre de projets outpput universiteacute id national -nombre de projets outpput universiteacute id national -valeur $ outpput id national
des deacutepenses de recherchebudget total des deacutepenses input universiteacute G10 national
classements internationaux outcome THES Shanghai Taiwan nationalinternational
RP gouvernance-reddition de comptes2008-07-08
- 1Premiegravere partie-Rapport sur les indicateurs de performance
-
- Preacutepareacute par Roland Proulx
- Consultant en planification institutionnelle et veille strateacutegique
-
- 2Inventaire des indicateurs
-
- inventaire
-
- 3ANNEXES
- 4Performance Indicators
- 5a Page couverture-A world of difference
- 5bTexte-AWorld-of-Difference
- 5ctableau0a word
- 6The Real World of
- 7gouvernance3
- 8gouvernance_reddition
-
- reddition
-