Talk:Training Metrics

(Transcluded from Talk:Education Committee/Education Committee meeting 28 May 2013) Please can the review result in measurements of key metrics from the program (for examples, see Project Metrics), so that the success of the program can be quantified rather than just qualified? Thanks. Mike Peel (talk) 09:07, 3 May 2013 (UTC)

The review is being carried out by the office, not the Education Committee. That is not to say that the point is unimportant. I'm going to post something here, for the world to see, and then take the point to an email discussion also.

By way of an answer: the question of metrics was aired in the meeting, but the discussion there wasn't exactly conclusive. Here is a personal take of mine, so we can at least debate something. This table is about numbers you could get from a typical workshop, in the context of some of the benefits a workshop could bring.

#	Type of advantage	Specifics	Quantifiable?	Comments and examples
1	Training	Increased editing of the projects by trainees	Y	The assumption is that trainees continue editing with the same account, and accounts are logged at the workshop
2	Training	Spread of know-how through an institution by a trainee	N
3	Training	More appropriate editing by a trainee	N	For example, if a trainee who is on the comms staff of an institution simply refrains from editing with a conflict of interest, in the Wikipedia sense, who is to know? But the outcome is positive for Wikipedia.
4	Training	Trainee considers workshop a contribution to their professional development	N	This the flip side of the previous point: the training might actually be advantageous to the trainee, and conventionally this is understood to be the case. One unquantifiable plus from the Midas training-for-trainers would be that this sort of point is featured, implicitly and explicitly.
5	Outreach	Wikimedia awareness	N	People generally have no idea of the various different meanings of "Wikimedia" (there being at least three)
6	Outreach	Referrals	More a question of logging than counting	For example: trainers are invited back by an institution. A trainer is invited to speak at a non-WMUK conference. A workshop results in a student project. (These are recent real-life examples.) Word-of-mouth from one institution to another.
7	Outreach	Returning customers	Hit and miss	Someone coming to a workshop goes to another, a conference, a meetup ...

Ideally before-and-after is the form of a review, from a baseline. In fact the training programme took shape fairly spontaneously, and I for one don't really see what the baseline would be. Charles Matthews (talk) 20:03, 31 May 2013 (UTC)

Thanks Charles - this is a good starting point to talk about this. I've added index numbers to your table - hope that's OK - and I've made some comments and suggestions below on each point.

1. I think this is the main thing to start quantifying, particularly because it's the easiest. Some of the metrics that could be measured here could be numbers of:

New people editing the projects

New people still editing the projects after [6 months, 1 year, 2 years]

Amount of content contributed by the new editors (text/images/etc.)

People that didn't continue editing (and qualitatively, why)

2. This is more difficult to measure, but there are some metrics that could be measured here. E.g.:

New institutions working with Wikimedia

Number of events taking place at institutions that have been organised by those that have gone through Train the Trainers

3. I'm not sure how this can be measured. Perhaps by a before-and-after survey of trainees to test their understanding of the projects?

4. This is something that could be measured by surveying trainers, which could yield qualitative results.

5. Perhaps a survey of trainees to check their understanding of the different Wikimedia projects? Figuring out the different meanings of 'Wikimedia' isn't really the main aim of increasing awareness of Wikimedia.

6. I'm not entirely sure what is meant here - is this a case of measuring repeat events and number of new opportunities arising, or something else?

7. If we use CiviCRM's events module to register people going to workshops, WMUK conferences, etc. then we could probably build up a good database here fairly easily. There's an interesting debate to be had about whether this is a more important metric than (1) or not. ;-)

Thanks. Mike Peel (talk) 20:47, 31 May 2013 (UTC)

Well, I would say #6 is particularly important to WMUK, and the fact that it doesn't look very "metric" mostly speaks to the fact that metrics are a Procrustean bed. But you'd better see what others have to say. Charles Matthews (talk) 21:13, 31 May 2013 (UTC)

I started page called Training Metrics and transcluded the chart their and this discussion to the talk page there. I think that although, as Charles points out, the review is to be carried out by the office, it is important that the Education Committee be fully engaged with this. I suggest we continue the discussion on the associated talk page.Leutha (talk) 11:53, 1 June 2013 (UTC)

I started page called Training Metrics and transcluded the chart their and this discussion to the talk page there. I think that although, as Charles points out, the review is to be carried out by the office, it is important that the Education Committee be fully engaged with this. I suggest we continue the discussion on the associated talk page.Leutha (talk) 11:54, 1 June 2013 (UTC)

That's not transclusion, but hey. I think you gents are all missing an important part of the impact of this programme. It's impact on the trainees trained by those who have attended these courses is of course the most important thing to try to measure because it's the raison d'etre of the program, but you need to also look at its impact on the trainers who have attended the courses. I think WMUK has more people who feel able and empowered to deliver training and who have delivered training than before we sent people on these courses and some of those people are becoming more involved in WMUK's work. I base this on anecdotal evidence, but I've worked with most of those who have delivered training since attending one of these courses, and I expect the statistics will back me up on the parts that can be quantified. Harry Mitchell | Penny for your thoughts? 16:06, 2 June 2013 (UTC)