Talk:Education Committee/Education Committee meeting 28 May 2013

Jon is facilitating the review of Train the Trainers, and has sent a message to the Education Committee on 1 May 2013 asking for input. The discussion is meant to he head via email, but if there is need for further discussion it could happen during the meeting. For reference I am copying the email below.

Train the Trainers

• We have now completed our first year of this programme. We need to reflect on its successes and challenges. Below I set out my intentions as to how to go about this but would appreciate any extra thoughts you might have.

Aim

• To review the programme to date, its progress, it value and do a simple SWOT analysis.

Methods

• Questionnaire of participants
• Interview with providers, i.e. Midas and Andy M and Charles Matthews

Issues to particularly consider

• Accreditation and its methods
• The use of the VLE and its next steps
• Location of courses and the future
• Future potential
• How we are using the trained people
• Economics of the programme

Outcomes

• Analysis of impact and potential impact.
• Recommendations for future progress or cessation.

Metrics

Please can the review result in measurements of key metrics from the program (for examples, see Project Metrics), so that the success of the program can be quantified rather than just qualified? Thanks. Mike Peel (talk) 09:07, 3 May 2013 (UTC)

The review is being carried out by the office, not the Education Committee. That is not to say that the point is unimportant. I'm going to post something here, for the world to see, and then take the point to an email discussion also.

By way of an answer: the question of metrics was aired in the meeting, but the discussion there wasn't exactly conclusive. Here is a personal take of mine, so we can at least debate something. This table is about numbers you could get from a typical workshop, in the context of some of the benefits a workshop could bring.

#	Type of advantage	Specifics	Quantifiable?	Comments and examples
1	Training	Increased editing of the projects by trainees	Y	The assumption is that trainees continue editing with the same account, and accounts are logged at the workshop
2	Training	Spread of know-how through an institution by a trainee	N
3	Training	More appropriate editing by a trainee	N	For example, if a trainee who is on the comms staff of an institution simply refrains from editing with a conflict of interest, in the Wikipedia sense, who is to know? But the outcome is positive for Wikipedia.
4	Training	Trainee considers workshop a contribution to their professional development	N	This the flip side of the previous point: the training might actually be advantageous to the trainee, and conventionally this is understood to be the case. One unquantifiable plus from the Midas training-for-trainers would be that this sort of point is featured, implicitly and explicitly.
5	Outreach	Wikimedia awareness	N	People generally have no idea of the various different meanings of "Wikimedia" (there being at least three)
6	Outreach	Referrals	More a question of logging than counting	For example: trainers are invited back by an institution. A trainer is invited to speak at a non-WMUK conference. A workshop results in a student project. (These are recent real-life examples.) Word-of-mouth from one institution to another.
7	Outreach	Returning customers	Hit and miss	Someone coming to a workshop goes to another, a conference, a meetup ...

Ideally before-and-after is the form of a review, from a baseline. In fact the training programme took shape fairly spontaneously, and I for one don't really see what the baseline would be. Charles Matthews (talk) 20:03, 31 May 2013 (UTC)

Thanks Charles - this is a good starting point to talk about this. I've added index numbers to your table - hope that's OK - and I've made some comments and suggestions below on each point.

1. I think this is the main thing to start quantifying, particularly because it's the easiest. Some of the metrics that could be measured here could be numbers of:

New people editing the projects

New people still editing the projects after [6 months, 1 year, 2 years]

Amount of content contributed by the new editors (text/images/etc.)

People that didn't continue editing (and qualitatively, why)

2. This is more difficult to measure, but there are some metrics that could be measured here. E.g.:

New institutions working with Wikimedia

Number of events taking place at institutions that have been organised by those that have gone through Train the Trainers

3. I'm not sure how this can be measured. Perhaps by a before-and-after survey of trainees to test their understanding of the projects?

4. This is something that could be measured by surveying trainers, which could yield qualitative results.

5. Perhaps a survey of trainees to check their understanding of the different Wikimedia projects? Figuring out the different meanings of 'Wikimedia' isn't really the main aim of increasing awareness of Wikimedia.

6. I'm not entirely sure what is meant here - is this a case of measuring repeat events and number of new opportunities arising, or something else?

7. If we use CiviCRM's events module to register people going to workshops, WMUK conferences, etc. then we could probably build up a good database here fairly easily. There's an interesting debate to be had about whether this is a more important metric than (1) or not. ;-)

Thanks. Mike Peel (talk) 20:47, 31 May 2013 (UTC)

Well, I would say #6 is particularly important to WMUK, and the fact that it doesn't look very "metric" mostly speaks to the fact that metrics are a Procrustean bed. But you'd better see what others have to say. Charles Matthews (talk) 21:13, 31 May 2013 (UTC)

I started page called Training Metrics and transcluded the chart their and this discussion to the talk page there. I think that although, as Charles points out, the review is to be carried out by the office, it is important that the Education Committee be fully engaged with this. I suggest we continue the discussion on the associated talk page.Leutha (talk) 11:55, 1 June 2013 (UTC)