Talk:WMUK membership survey 2013/Survey draft

Demographics

Ok so in 2012 we asked questions about income, marital status, and children. What we were trying to do was get a sense of barriers to attend events or the circumstances of members households so we could be accommodating (offering a creche at a conference maybe!) However I'm not sure these questions are as useful as we hoped.

What are better questions to ask that cover important problems like:

• How affordable is attending events in terms of travel and fee costs?
• Do your household circumstances present a potential barrier to participation? i.e. young or single parenting, carer, student under age of 16 etc

I think we don't need to ask about people's relationship status - its not useful or necessary. It could be helpful to ask about their households, but again I'm fuzzy on why. Best not if we're not sure. Katherine Bavage (WMUK) (talk) 10:01, 16 October 2013 (UTC)

Those are some really personal questions. If I was asked that in a survey from an organisation I was peripherally involved with, I wouldn't answer them, and I probably wouldn't respond to the survey at all. Do we really need to know if somebody is transgender, and if we do, do we really need to know what their gender was at birth? How does that help us? The same goes for sexual orientation, annual income, and relationship status. I can understand asking whether members are parents/carers, but do we really need to know how many children they have? There are also carers of disabled adults to consider. If there's something specific you want know - eg whether affordability is a barrier - why not ask that instead of making assumptions? Harry Mitchell (talk) 13:41, 16 October 2013 (UTC)

Thanks Harry. I'll try and break it down a bit cos you've raised a few points;

• "Do we really need to know if somebody is transgender, and if we do, do we really need to know what their gender was at birth?" - This is following best practice guidelines about asking questions about gender identity. Basically asking 'Are you a 'man', 'woman' or 'other' is just not good enough (NOT that I'm saying that's what you're suggesting.) If you look at what I was talking about on the intro page and the draft blurb for the demographics section, the reason you might monitor gender identity is both to do with trying to monitor if our membership is broadly mirroring that of wider society, and if not are there things we need to think about in terms of outreach work or barriers to participation. I'm really hoping this kind of data will feed into what people bring back from the diversity conference for instance. So that's the 'why'
• "I wouldn't answer them, and I probably wouldn't respond to the survey at all." - yep that's definitely a risk. I have on my to do list to do some reading about how to develop/encourage ways organisations can encourage people to disclose this information because they can see the value it will create. Again, this will be anonymous, all the questions will be optional as well, and I hope if we can link the ones we DO ask to data that informs decisions that might demonstrate value to people.
• "If there's something specific you want know - eg whether affordability is a barrier - why not ask that instead of making assumptions?" It's not that simple. You get responder bias about how relatively well off people are, and you cant compare that to office of national statistics data about national or regional trends we can benchmark against. I would welcome suggestions for better questions, and I'd probably err on the side of not including questions than asking rubbish ones. I like asking about caring commitments rather than number of children per se for example.
• Thanks so much for feedback. If you can make suggestions for questions that would help me lots, or just come and review what I've worked on next week IF you get chance (I know you're busy :-)) Katherine Bavage (WMUK) (talk) 13:57, 16 October 2013 (UTC)

Additional note - we should be asking about disability in this context too. Katherine Bavage (WMUK) (talk) 14:26, 16 October 2013 (UTC)

Good points. As I think I commented, I copied the text for the demographics questions from the best practice page on meta (as I can't think of a reason to not follow best practice). As for what to ask, my thought was to draft lots of potential questions and then select the ones we want to ask from that list. Thryduulf (talk: local | en.wp | en.wikt) 22:03, 16 October 2013 (UTC)

I too find the questions in the demographics section far too intrusive and too remote from the projects, in particular 2,3,4,7,8.9 and 10. I would normally bin any survey that asked such questions. We are in danger here of following so much best practice that we get the result that most of these surveys get of almost no replies. Following recent data protection lapses that have been covered in the media I think that most people have little confidence that their data will be kept confidential and it might be better to adopt a light touch with this survey. For gender, why not just ask people what gender they are and leave it at that? Let them fill in the reply free form and drop questions 2-4. Asking about relationship status and children also seems to go too far and it's hard to see why we need to do so. Honestly, what business does WMUK have asking about these things? Philafrenzy (talk) 18:48, 24 October 2013 (UTC)

Incidentally, Schedule 1 to the Data Protection Act states that sensitive personal data should not be processed unless it is "relevant" and "not excessive" in relation to the purpose for which it is processed. Philafrenzy (talk) 11:49, 25 October 2013 (UTC)

My supposition is that we are asking this information so that we can use it to compare our results with the population as a whole and with participation in Wikimedia projects as a whole. Ottomh I'm not sure why a distinction between married/civil partner/living with someone as if they were married is relevant though. Re gender, there was criticism from transgender groups about the Wikipedia data on transgender participation as there were only options "male","female","transgender" which they think meant that the number of transgender people was underreported as a transman for example would want to tick "male" and "transgender". What reposnsibilities people have for caring for children is important for planning of events and understanding barriers to participation at events. You have just given me the thought that we should explicitly ask whether the availability of childcare is a barrier to WMUK events, or whether we should be holding meetups in more child-friendly locations than pubs. Thryduulf (talk: local | en.wp | en.wikt) 12:07, 25 October 2013 (UTC)

Hi Philafrenzy. We had the same objections last year and we're going to move to a 'light touch' in the sense of the demographic questions a) Coming last b) Being optional and anonymous c) Having a clear statement on use, storage and purpose. I don't want to throw the baby out with the bathwater however - having NO questions about who are members are leaves us in a poor position to serve their needs.

It would be very helpful if people suggested better questions as well as saying they don't like existing ones :) I agree that asking if someone is married is basically irrelevant so better questions about circumstances that affect participation and engagement needed, but also ones that are somewhat objective (so not 'can you afford to attend events' because this tells us nothing useful about how we could help) Also if you have come across survey experts across the projects over the years do point them in the direction of this discussion and the meta page because I think we could all do a bit better on this front. Katherine Bavage (WMUK) (talk) 13:30, 25 October 2013 (UTC)

Can everyone drop in questions you would expect to have answered in here so I can try and factor that in? WMUK_membership_survey_2013/FAQs Katherine Bavage (WMUK) (talk) 13:53, 25 October 2013 (UTC)

The intro to the demographics sections currently includes the words "we want to reach out to less-well-represented groups and engage them.", would that be better phrased as "...engage with them"? Thryduulf (talk: local | en.wp | en.wikt) 21:19, 25 October 2013 (UTC)

How about a plain English re-write? That section seems packed with jargon, we don't have to inherit the management-speak from the WMF. As for 'engage' it reminds me of engaging the Borg. --Fæ (talk) 00:18, 26 October 2013 (UTC)

Please feel free to have a go :) Katherine Bavage (WMUK) (talk) 11:32, 26 October 2013 (UTC)

By all means ignore these comments if there are no resources to improve the survey this way.

I spent a lot of volunteer time writing up policies when we created the charity, being one of the few people around who had suitable qualifications and experience. Now there are people paid full time to do it, writing and improving this stuff for free seems far less worthwhile compared to using my wiki-time to add another 100,000 images to Commons. Happy to consider being paid should that ever arise. --Fæ (talk) 12:09, 26 October 2013 (UTC)

Yes you did and its how I know you're a good writer :) If you're too busy I understand - its just that its hard for me to rewrite what I wrote in a way you want - I thought it might be easier for you to just make some changes. Anyway, I'll have look at trying to split up sentences and simplify words - and anyone else reading this who could do that, thanks! Katherine Bavage (WMUK) (talk) 13:03, 26 October 2013 (UTC)

PS perhaps if we linked to engage the borg videos everytime I use the word engage this would dramatically increase participation :-) Katherine Bavage (WMUK) (talk) 13:05, 26 October 2013 (UTC)

I am unclear how some of the responses gathered in this survey will be used by WMUK, particularly the gender ones but others too such as income and education. If we gather personal data the DPA requires it to be for a clearly defined purpose and at the moment the purpose seems to be mainly that we would like to know rather than because we have any practical use for the data. For instance, if we discover than 10% of the membership are educated to A level standard, or are asexual or are aged 25-34 what exactly can we do with that information? It's not like you can hold an editathon targeted at asexual 30 year-olds is it? Would we have one newsletter for people educated to degree level and another in simple English for everyone else? There is also the matter of the statistical significance of the results. Even if we get 100 replies rather than the circa 50 last year, some of the results (2 or 3 ticks) simply won't be statistically significant and for that reason unusable in any practical sense. I think we should focus in solely on questions where there is some practical application for the data gathered with each question or set of questions requiring a rationale for inclusion. Philafrenzy (talk) 15:40, 27 October 2013 (UTC)

Part of why we want to know is that people (including the WMF I suspect for things like gender) ask and expect us to know. Our objects include "...to promote and support the widest possible public access to, use of and contribution to Open Content...", and to make sure we are doing that we need to know who we are reaching. Members are obviously not the only people we reach, but they are a component of that. Additionally, it is perhaps more useful to know which groups of people are under/not represented rather than those who we are. So if 90% our members are male then we need to focus on reaching more women. Equally, if our membership is significantly more representative of a particular group than the WMF as a whole then it is worth identifying what we are doing well so that others can benefit from our experience. 16:57, 27 October 2013 (UTC)

OK, but that does rather confirm my point that we are asking these things because other people expect us to rather than because there is anything much we can actually do with the results. Because of the sample size the only reliable aspect of the gender question will be those who identify as male or female as the numbers ticking the other options, based on the previous survey, are likely to be so few as not to be statistically significant (no offense intended!). I still don't know what exactly WMUK can do with the information apart from file it as it is not clear to me quite how somebody being trans, for instance, has any material effect on their ability to attend a meetup or edit any of the projects which are essentially intellectual, not social, in nature. Please explain what I am missing. Philafrenzy (talk) 17:28, 27 October 2013 (UTC)

I think it is not a good sentiment to suggest that because a group might be small, including them in a survey is unimportant. In fact I really object to the idea underneath that.

I have repeatedly made the following point; we need to ask a question about gender. In order to do that, I believe we should include responses that acknowledge different gender identities. No one has actually argued that is incorrect. Philafrenzy, is that what you are arguing i.e. that we don't need to ask a question about gender? I would obviously disagree. We an not diverse organisation and we need to monitor efforts to improve that.

What we do with the data - first of all reporting it is important because it keeps us on our toes when we're not good enough at something. Next - I assume that following the report staff, the board and the community would want to make decisions based on what it tells us in terms of outreach or other elements. Do we need some unconscious bias training, for example.

"it as it is not clear to me quite how somebody being trans, for instance, has any material effect on their ability to attend a meetup or edit any of the projects which are essentially intellectual, not social, in nature." This does not seem to acknowledge the considerable research and reporting on the barriers people with different gender identities may experience trying to get involved. It is imporant to understand that just because we, personally, cannot see a problem this may be because we are different not because there is no problem.

TL:DR - Can we get back to: should we have no questions on gender, or questions that acknowledge different identities, and if the latter, what should these be? Katherine Bavage (WMUK) (talk) 10:54, 28 October 2013 (UTC)

Katherine, I certainly didn't say that any group should be excluded because it is small, on the contrary, I am arguing that we should treat people as complex individuals and not try to fit them into ever finer categories based on income, education, gender or anything else. It really is insulting to people to ask them to tick a box to say what particular category of orientation they fit into as if such things are permanently fixed at birth, particularly when we have no clear idea of what we will do with those results. People want to be acknowledged for their individuality. We are not statistics, results in a survey or "groups" to be targeted. Why don't we stick to asking about what we can do for people and how we can do it better rather than trying to fit everyone into neat statistical categories? Philafrenzy (talk) 12:17, 28 October 2013 (UTC)

We may just have to respectfully agree to disagree, as you appear to principally opposed on monitoring any characteristics. I have discussed this with other users before - why *do* we care when in a perfect world we approach people as contributors not a series of labels? With respect I would suggest there is enough evidence within the movement that this is not what is happening and that certain groups feel excluded. In order to solve this we need to demonstrate any attempt to improve matters is having an impact. Not monitoring is contra the approach of much of the wider movement, and indeed, probably a good idea for a registered charity with outreach as part of its public object.

I do sincerely value your contribution and opinion though so thank you for this, even if we do disagree on this point I know you have been passionately interested in recruiting members and getting our numbers up :) Do have a look at the other sections as well to make sure they reflect some of the ideas/problems you've identified Katherine Bavage (WMUK) (talk) 12:29, 28 October 2013 (UTC)

The question order

Probably sucking eggs here.

General practice in such questions is to start with the most general questions and then move towards the specifics. This launches straight into questions that affect very small parts of the population. Should start with ethnicity, identified gender, age, etc and then work towards more specific things.

We rightly make sure that people realise such questions are optional. Jon Davies (WMUK) (talk) 10:48, 16 October 2013 (UTC)

The demographics questions will - in many cases - put people off straight away. They should be moved right to the end, I think. --MichaelMaggs (talk) 18:11, 16 October 2013 (UTC)

A few months back I spent a while filling in lots of professionally developed surveys about all sorts of things. From memory they were split about 60/40 asking demographics questions first/last. I don't think that provides strong guidance either way, so unless someone knows of a reason why they should be first moving them to the end would seem fine to me. The order of sections on the draft page reflects a combination of the order they were asked in last year (I don't know how that was decided) and what seemed logical to me at the end of the day yesterday.

I believe that all the questions will be optional, and that this will be clearly stated in the final presentation (on SurveyMonkey). Explicitly saying that the demographics questions are optional shouldn't be too difficult, but as we want people to answer them (otherwise we wouldn't ask) so we don't want to discourage people so some thought should go into the warning of any such marking.Thryduulf (talk: local | en.wp | en.wikt) 22:11, 16 October 2013 (UTC)

I'd agree that they should be moved to the end. The survey should start with the most important questions that are being asked, in order to maximise participation in that part of the survey. Optional sections such as the demographs should be last so that people that don't want to answer them can easily say 'no' and submit the rest of their responses, and also so that they aren't put off filling in the survey at the first page. If people have already spent time filling in most of a survey, and are then presented with a section they don't want to fill in, then they're more likely to submit their responses and use the option to skip the rest than with someone that is asked if they want to skip optional demograph questions right at the start without having already invested their time in the survey. Thanks. Mike Peel (talk) 22:31, 22 October 2013 (UTC)

Attendance at events

I've added a question to the draft about whether respondents have ever attended an event. I think it would be worthwhile to ask some more detailed questions about barriers to attendance (such as not knowing about events, no events in their area, being unavailable at weekends, etc) and whether addressing some of those might make them more inclined to come to an event. It might also be worth asking whether respondents have edited Wikipedia (and if no whether they'd like to) to see if we et any correlations to help make sense of the data. We could also ask how they first heard of WMUK, and it might be nice to ask for suggestions for future events - most of the responses might not be of much use, but we might get somebody who works for an organisation we want to partner with and is willing to act as a go-between for example. Harry Mitchell (talk) 14:04, 16 October 2013 (UTC)

Good plan. We did this a bit last time but it over focused on meetups. I'll have a go at drafting some questions :D Katherine Bavage (WMUK) (talk) 14:09, 16 October 2013 (UTC)

We should also ask about projects other than Wikipedia. Thryduulf (talk: local | en.wp | en.wikt) 22:12, 16 October 2013 (UTC)

"One of the main activities that WMUK organises are offline events." Does this read a little strangely to anyone else? I think it's the "one of...are", but I'm not sure and I can't think of a better way to phrase it myself. Thryduulf (talk: local | en.wp | en.wikt) 20:11, 25 October 2013 (UTC)

It also sounds odd to me. I've suggested "main types of activities" - does that work better? Thanks. Mike Peel (talk) 19:48, 26 October 2013 (UTC)

Not perfect but much better. Thryduulf (talk: local | en.wp | en.wikt) 21:46, 26 October 2013 (UTC)

Gender questions

Why would we want to ask "If you are transgender, what was your assigned gender status at birth?". As the previous questions asked for gender identity and whether you are transgender, this extra detail seems oddly unnecessary and possibly offensive, particular if asking someone who was born intersex to fill in "Other (please specify)". I suggest this particular question is dropped as problematic.

Anyone reviewing this might find GLAAD's guide helpful.[1] --Fæ (talk) 17:01, 24 October 2013 (UTC)

Yes, drop it and a few others too. Philafrenzy (talk) 18:48, 24 October 2013 (UTC)

The question is copied verbatim from the best practices page on meta, which sites http://gaygenderresearch.org/ as the source for that question. It seems they have reorganised their website since the page on meta was written. I don't have time to hunt for the new location right now, sorry. Thryduulf (talk: local | en.wp | en.wikt) 11:56, 25 October 2013 (UTC)

If someone can track it down it would be useful to compare with. I suggest a key criteria for deciding which questions to ask in a WMUK survey should be that "if we don't understand and publicly explain how we would use specific personal data, then we don't collect it." With Wikipedia in the news with regard to poor treatment of transgender subjects, it would be better for WMUK to be criticised for excessive caution or political correctness, rather than a lack of care. --Fæ (talk) 12:42, 25 October 2013 (UTC)

For my twopennoth - this isn't my preferred approach to asking questions about gender - this is. As to why we might ask questions about gender - I'm guided by the twin principles of understanding and better serving a respondee group and in this case, asking questions about gender that represent a range of circumstances. It might seem tempting to just stick to asking if someone is 'male' or 'female' or 'transgender' but that's actually quite poor practice (I'm not suggesting that is what people have recommended, just explaining why a superficially simple solution isn't necessarily a good one)

Please be reassured I understand that in doing so the questions would have to be anonymous and optional - if you look elsewhere you'll see that we effectively plan to have the demographic questions as a second 'follow up' survey and encourage buy-in by demonstrating clearly how the data will be safe and could inform decision making. I can elaborate on that?

What is people's instinctual preference here? I think we have to choose between not asking about gender at all, or asking in a representative and accessible way. I personally prefer the latter but am happy to listen to views. Katherine Bavage (WMUK) (talk) 13:23, 25 October 2013 (UTC)

It's not instinct, rather understanding what we are going to do with the result and whether the outcome is then worth the burden of the question. Could you explain why we would want to know "assigned gender status at birth" if we already have asked if someone identified as transgender and they already gave their gender identity?

Considering that someone who might actually give an answer to the question has probably spent a long time moving on from how they were labeled at birth, and WMUK is neither a LGBT, gender or political specialist organization, this level of detail seems both intrusive and superfluous. GLAAD's general principle for journalists is to focus on current gender identity rather than making a big deal out of past or biological gender, as indeed is the evolving standard for Wikimedia (as far as I can tell, this was the whole point of the recent fracas at en.wp).

Oh, by the way, the survey may be anonymous, but we have very few members. This means that if only 50 people reply, and of these the stats say that only 5 people identify as LGBT and only two say something about being transexual, well, it's not that hard to work out who they must be, particularly for me with a background with Wikimedia LGBT. For these reasons any precautions of anonymity might be undermined by basic deduction. --Fæ (talk) 15:03, 25 October 2013 (UTC)

OK so to clarify the thinking behind the current questions

1. If we ask questions about gender, should they be representative of the lived transgender experience or just 'male'/'female'(Not other - I didn't like that question set and changed it yesterday)
2.  I think we need to understand the gender balance of our membership because it is still predominantly male and we need to address that. I hope that clarifies what we are going to do with the result - it will be reported feed into thinking across what the charity does and hopefully improve representation of groups that current have low engagement.
3. I don't think that the best way to do this is to effectively not allow people who identify as transgender to have that represented. Could you confirm you've had a look at the Equality and Human Rights commission guidance that I linked to? That explains why this might not be a good approach. I'd rather you refute that so I know we're operating from fundamentally different perspectives rather than a question of tweaking the survey draft. I'm actually not sure we *are* so far apart - the GLADD guide (another good one by the way, thanks) clarifies

   'When describing transgender people, please use the correct term or terms to describe their gender identity. For example, a person who is born male and transitions to become female is a transgender woman, whereas a person who is born female and transitions to become male is a transgender man.'

   If this is the case are we in consensus that we could ask questions, but better ones (and what would these be?) or do you maintain we just shouldn't ask because we're a generalist org not a representative one?
4.  The question of numbers/anonymity. If someone filled out this form then the person reviewing the response could possibly cross calculate characteristics to identify the person (E.g. This person is born in 1970, is male, and says their ethnicity is white irish - oh that must be Bob) That is always an issue when the pool of people you are surveying is small as you say. However - back to first principles - does that mean you don't survey small groups? In my opinion, no, it means you undertake the following precautions; a) All responses are anonymous b) All questions are optional c) Only a limited number of analysts review the individual responses as opposed to the meta data d) The meta data is reported according to best practice (again referring to guidance - '*'ing population numbers if the numbers are below a certain level) e) The individual responses are deleted after they have been reviewed f) a - e are clearly communicated to all participants.

I hope this is helpful. To help me it would be good if you have a look at the Equality and Human Rights commission guidance and then feel free to come back and tell me where this doesn't square with our community approaches as you refer to and I will definitely consider it. I'm in complete agreement about prioritising the needs of respondents and the burden of the question being worth the outcome. That is very important and I would probably rather reluctantly have no demographic data on members than cause an individual worry or embarrassment from clumsy questions. Its only because I think trying to have a sense of gender identity amongst members might help lead to better representation in the long term that this section matters. Anyway - value your input to make the right call even if we are approaching this from different angles. Katherine Bavage (WMUK) (talk) 12:06, 26 October 2013 (UTC)

With regard to the section title here, I was specifically raising the transgender questions as an issue, rather than wanting to get drawn into a more general discussion about surveying and analysing gender.

I did look at the Equality and Human Rights commission guidance, I don't need to look at it again, and it just did not seem pitched for the organization that WMUK is. That was the point I was making previously.

In terms of the current questions with regard to transgender, they still seem superfluous to anything the charity would actually do with them, if anyone answers the questions this becomes a risk if the data is used and reported in any form as the numbers would be so low as to reveal unnecessary personal information, and I don't understand how this is particularly helpful in delivering the mission of the charity. For example I don't know of anyone who was "officially" declared as intersex at birth, that is incredibly unlikely to have happened any date before the current century hence irrelevant to the target sample space, and the questions about "the process" of changing sex are intrusive verging on offensive, considering I cannot imagine why WMUK would want such information from the one or two people that might answer it.

Personally, I have spent about 2 hours on this, in contrast this week I have spent around 8 hours helping with a mass upload on the Welsh Wikipedia and around 20 minutes on a harassment problem on IRC. Having expressed my opinion fairly clearly on the record already, on balance I don't think that spending a long time refining one or two questions that extremely few people will give an answer to is worth diverting much more time on. If you would like more relevant alternative opinions to balance mine, I suggest raising a request for views on what is suitable at m:LGBT. Thanks --Fæ (talk) 13:25, 26 October 2013 (UTC)

Raising at m:LGBT is a good idea and I will do that - thank you :) Yes quite agree you might have had enough so thank you for your input and I am listening :) Good work with the Wicipedia uploads and supporting people too Katherine Bavage (WMUK) (talk) 13:57, 26 October 2013 (UTC)

I've made a suggested change to the page along the lines of what I would view as a simple and neutral approach here - saying that you are male or female (which can be done irrespective of birth gender), identifying as transgender and specifying that you identify as male of female now (again, this is irrespective of birth gender), identifying as asexual (neither male of female), or 'Prefer not to say'. Does this cover all bases, or have I missed an important aspect here? Please feel free to revert my change here if this isn't appropriate. Thanks. Mike Peel (talk) 20:59, 26 October 2013 (UTC)

I've added an "other" option to your suggestion as that way we don't exclude anyone. Given then other options I don't think it is likely that anyone will tick it, but gender identity is complicated and its best not to exclude anyone. Thryduulf (talk: local | en.wp | en.wikt) 21:38, 26 October 2013 (UTC)

Can we change it to at least 'Other (Please specify) and a free text' so it's more about people feeling they could provide the most accurate description of their identity, rather than a sort of catch all/derisive 'none of the above' option? Anecdotally I believe 'other' as an option is not always considered respectful :/ Katherine Bavage (WMUK) (talk) 10:59, 28 October 2013 (UTC)

Yes, absolutely to "other (please specify)". As for respect, I hadn't thought about that but you are right. The options are though to list everything possible (long, and we run the risk of missing something), "other (please specify)" (potential respect issues) and "[other (please specify) using more respectful wording]". If we can comeup with respectful wording I think that will be the best option? Thryduulf (talk: local | en.wp | en.wikt) 11:35, 28 October 2013 (UTC)

I have raised the particular point about asking questions on gender identity on m:LGBT as suggested. You can view the post here Katherine Bavage (WMUK) (talk) 11:22, 28 October 2013 (UTC)

Just a thought about age. As far as we know are there members under 18? It might not be appropriate to ask some of these questions of children. Philafrenzy (talk) 00:44, 31 October 2013 (UTC)

If this were a charity that specialized in supporting minors with gender dysphoria or children with problems at home due to being open about their LGBT identity or sex life, then I could see why we could justify collecting this sort of information. As someone interested enough in this topic to set up Wikimedia LGBT, and as said earlier, I have yet to understand why WMUK wants to collect this personal data, particularly when data on sub-groups such as minors or trans people will become an obvious risk of outing people in one form or another. You could count the number of minors that come to our events on the fingers of one hand, having information on them about whether they were intersex at birth or whether they identify as gay seems an unnecessary risk for the charity as well as the volunteers if the data is ever used, through whatever process of anonymity. These issues have eaten up far too much employee and volunteer time compared to any value the questions might have and the relatively tiny sample space this survey will have available. --Fæ (talk) 07:03, 31 October 2013 (UTC)

This could be avoided by moving the age question to the start of the section, and only asking the latter questions if they select an age of 18 or above. Thanks. Mike Peel (talk) 10:13, 31 October 2013 (UTC)

Yes that's doable - good suggestion.

Fae, I'm afraid I don't agree with you about the risk of outing following the safeguards I outlined above which are:

a) All responses are anonymous b) All questions are optional c) Only a limited number of analysts review the individual responses as opposed to the meta data d) The meta data is reported according to best practice (again referring to guidance - '*'ing population numbers if the numbers are below a certain level) e) The individual responses are deleted after they have been reviewed f) a - e are clearly communicated to all participants.

I have also already said I do not agree that only specialist organisations should monitor diversity in their membership or composition.

None of the respondees who have contacted me privately about these questions following my email to the community mailing list and talk page on m:lgbt have expressed concerns and indeed the feedback has been about how to re-write the question to meet their needs and expectations rather than not asking it - it will be re-drafted again to reflect this. Given that we both appear to be repeating ourselves may I suggest that as I said to Philafrenzy; we may just respectfully have to agree to disagree and I hope I can demonstrate that effective safeguarding and reportage are worth it.

Exactly how many people contacted you privately? I am interested as it makes a difference if twice as many people felt the need to contact you in confidence as to responding on this talk page, then consulting members on this talk page will seem like the wrong forum. However if only half the number contacted you in private, then this would not be a good reason to overrule issues raised here. --Fæ (talk) 16:42, 4 November 2013 (UTC)

On a side note. I don't understand how your preferred approach here (not asking these questions) will help with the points you raise on the watercooler about improving/affording higher priority to LGBT outreach or focused work. Given that our membership is the pool of people from which Trustees and committee members and grant applicants are drawn it is important to know where we are unrepresentative AND prioritise approaches that will improve this, and I don't think we can do that by assuming things. Katherine Bavage (WMUK) (talk) 16:26, 4 November 2013 (UTC)

So to be clear about the implementation, if any question that has fewer than N replies will not be reported on, what will N be? With regard to my preferred approach, I have expressed a view on the specific questions and would prefer to drop some of them, especially the overly intrusive ones, I have not proposed an alternative approach and at no time have I suggested that WMUK ask no questions related to LGBT. With regard to issues raised on the watercooler, I don't really see any statistics coming from this survey helping with getting on with basic outreach to the LGBT community, which should have already happened regardless of surveys of members as it was a key line in our published and agreed Activity Plan. --Fæ (talk) 16:42, 4 November 2013 (UTC)

It would be most helpful for me if you just edited the questions with your suggested alternatives, or specified which you would delete and why. Please do, then we can disagree on specifics, bearing in mind feedback I've had from elswhere Katherine Bavage (WMUK) (talk) 17:07, 4 November 2013 (UTC)

Can I at least have a sense of what feedback you have had? I am finding this sort of secrecy bizarre, particularly if from folks on my WM-LGBT network. --Fæ (talk) 17:59, 4 November 2013 (UTC)

While I have no idea of who has contacted Katherine, I do know that the request was highlighted on at least one Facebook group dealing with transgender issues, that message specifically requested comment to Katherine by email (for anonymity reasons) and didn't mention the wiki. Thryduulf (talk: local | en.wp | en.wikt) 22:33, 4 November 2013 (UTC)

I asked people to comment publicly or contact me privately and they chose to do the latter. Its not really appropriate therefore for me to share details about who has contacted me privately or what they said. You will, I'm afraid, have to take it on trust. If you find that difficult I'm at a bit of a loss as to how to reassure you given the concern about outing people. I will note I have redrafted the questions on gender identity and sexuality specifically based on that feedback. You are, as ever, welcome to encourage anyone you know personally to comment publicly on this issue where it is directly relevant to them. Katherine Bavage (WMUK) (talk) 11:00, 6 November 2013 (UTC)

Hm, secrecy = trust, okay, then I guess that due to the way you have chosen to run your review, I am not able to ask any more questions about this. --Fæ (talk) 15:58, 6 November 2013 (UTC)

Yes. To clarify. I chose to allow people affected by questions of gender identity to contact me privately so they could express an opinion without being outed. This is pretty reasonable. I have no idea what else you would suggest I do but I need to close the draft on this before Friday, so I think we're out of runway. As I have repeatedly said; if you have concrete suggestions for alternative questions please put them in and I can make a judgement call on whether they are better. Katherine Bavage (WMUK) (talk) 16:22, 6 November 2013 (UTC)

Fæ, you cannot have it both ways. You are adamant that we must allow people to contribute anonymously, and then you are complaining that people have chosen to do so. Can you not see the contradiction in that? Thryduulf (talk: local | en.wp | en.wikt) 16:31, 6 November 2013 (UTC)

"restricted reference material"

What does Q9 mean by "Access to restricted reference material", specifically what does "restricted" mean in this context? I'm guessing it's nothing to do with security classification? Thryduulf (talk: local | en.wp | en.wikt) 12:49, 25 October 2013 (UTC)

In the context of that question, I would think that mean restricted to members. -- Katie Chan (WMUK) (talk) 13:11, 25 October 2013 (UTC)

Actually I think Mike may have meant journals you pay to access or books you can't get hold of without buying or travelling to read? Katherine Bavage (WMUK) (talk) 13:24, 25 October 2013 (UTC)

Yup, I meant it to mean what Katherine's explained. Thanks. Mike Peel (talk) 15:48, 25 October 2013 (UTC)

Would perhaps "restricted-availability" express that better? Thryduulf (talk: local | en.wp | en.wikt) 19:32, 25 October 2013 (UTC)

Probably not, as it doesn't make it clear who's restricting the availability. I've suggested "Access to reference material that is normally behind a paywall" instead, although that doesn't cover material that isn't behind a paywall. Perhaps we need a point along the lines of 'Ability to request reference material by post' or a reference to microgrants for purchasing reference books? Thanks. Mike Peel (talk) 20:02, 26 October 2013 (UTC)

Intro

In the intro the sentence about how long the data is kept isn't brilliantly worded. Currently it reads:

When the meta report about the results has been produced and any responses that can be followed up stored on your member contact record, all responses will be deleted within 60 days of the survey closing.

I think either

When the meta report about the results has been produced and any responses that can be followed up stored on your member contact record, all individual responses will be deleted. This will be within 60 days of the survey closing.

or

Within 60 days, when the meta report about the results has been produced and any responses that can be followed up stored on your member contact record, all individual responses will be delete.

would be better. I would simply make the change, but I can't decide which I prefer. Thryduulf (talk: local | en.wp | en.wikt) 19:30, 25 October 2013 (UTC)

I don't object as long as its clear to readers when the data will be looked at, reported on and then deleted :) Pick a reword that you feel is clearest! Katherine Bavage (WMUK) (talk) 11:33, 26 October 2013 (UTC)

Both options sound like they would work well. I've also added a link to SurveyMonkey's privacy policy in this paragraph - it would be good if the corresponding WMUK privacy policy could be linked to here as well. Thanks. Mike Peel (talk) 19:51, 26 October 2013 (UTC)

I've gone for the first of the options I suggested above, but do feel free to change it again. Thryduulf (talk: local | en.wp | en.wikt) 21:51, 26 October 2013 (UTC)

NUTS regions

While I can see the logic in the concept of using NUTS, I'm not sure that they are the best structured for the purposes of locating events. Especially in Wales they do not correspond at all well to transport routes - Swansea or the Valleys to Cardiff is dead simple but they are in different regions, whereas Swansea to Caernarfon or Cardiff to Wrexham are intra-region but not easy or quick transportationally. Wales is best divided into South, Mid and North following the Severn Tunnel-Pembroke/Milford Haven, Shrewsbury-Aberystwyth and Chester-Hollyhead railway lines. In England the South East, South West and North West regions are too large to be meaningful - it's quicker and easier to get from Bristol to London than it is to get from Bristol to Cornwall for example (Dover is closer than Penzance!). Similarly London is much easier to get to from Oxford and Canterbury than it is to get between them. Thryduulf (talk: local | en.wp | en.wikt) 20:27, 25 October 2013 (UTC)

Urgh I know but it's hard to find a consistent measure that isn't really, really granular (i.e. at the counties level). Also we could ask about 'distance from home' but people are more likely to be able to travel cheaply to London than some places within 50miles of their home. I really don't know what the best solution is :( Katherine Bavage (WMUK) (talk) 12:42, 26 October 2013 (UTC)

Perhaps we could ask which public transport networks members have ready access to (e.g. west coast main line), which could then help inform the best locations for events? Thanks. Mike Peel (talk) 19:53, 26 October 2013 (UTC)

I think the NUTS regions give us a general idea. There will be anomalies like those whatever measure we use—I live in Derbyshire for example, but the Nottinghamshire border is a two-minute walk away, and within half an hour, I can be in the West Midlands or South Yorkshire, which are in completely different regions, yet parts of the East Midlands are a good couple of hours away. The only alternative I can think of is asking for the nearest city (or possibly major city), but a recent email thread suggests that note everybody knows their nearest 'major' city offhand. Harry Mitchell (talk) 13:36, 27 October 2013 (UTC)

Yeah it's a blunt tool basically.

On the public transport network thing - I'm still not convinced mostly because I have ready access to a lot of options in London but affordability would be my deciding factor to travel. Still a bit of a head scratcher this one. I'll try and make some time to scout out if there are clever ways other organisations survey this as a barrier to participation.

As much as I think in terms of accessibility in public transport terms, there are people who only think in terms of where they can drive. They wouldn't have a clue what their nearest main railway line is called, and names like "west coast" aren't exactly intuitive for people who only use it to travel from Coventry to Milton Keynes.

I had a quick look for travel based regions of the UK, but the only thing I found was travel to work areas, which are way too fine grained for our purposes. I suppose a first step would be to identify national organisations that deal with groups of people at local events. English Heritage and National Trust are limited by property locations and so use the English regions. Cadw use North/South/Mid/West Wales and Anglesey but other than the latter I can't find what the actual regions are. I haven't been able to find out whether Historic Scotland even use regions. Maybe the political parties or trade unions would be useful resources? Thryduulf (talk: local | en.wp | en.wikt) 13:13, 28 October 2013 (UTC)

Whichever regions we end up using, I think it would be useful to have some indication of which are more convenient for people than others. For example London is most convenient for me but the south east is also accessible. The easiest way to do this I suspect will be two columns, but I'm not sure how to label them. Possibilities for the first column I can think of are "most convenient", "1st preference", "definitely", "probably", "very convenient", "preferred", "most accessible", "easily accessible" with options for the second column "also convenient", "less convenient", "accessible", "also convenient", "also accessible", "possibly", "less preferred" and similar. Both columns would still be multi choice, as some people live on the borders of regions (e.g. Harry by the sounds of it). Thryduulf (talk: local | en.wp | en.wikt) 15:29, 30 October 2013 (UTC)

Disability and accessibility

I've added a question on disability. I couldn't find any simple question explicitly described as best practice (only for entire surveys about disability, which is far more detail than we need), so I've used the question from the 2011 census http://www.ons.gov.uk/ons/guide-method/census/2011/the-2011-census/2011-census-questionnaire-content/2011-census-questions---england.pdf It doesn't give us much detail, but I don't really think we need any more in this part of the survey - access needs, etc. are dealt with in part 1.

Should we ask in part 1 though whether the WMUK wiki and/or hardcopy materials are accessible? If we do ask I think we should have a free text box for suggested improvements. Thryduulf (talk: local | en.wp | en.wikt) 12:37, 26 October 2013 (UTC)

Yes to adding the free text for suggestions and thanks for adding the questions - we need to make sure these are good and robust :-) Katherine Bavage (WMUK) (talk) 12:40, 26 October 2013 (UTC)

I've found http://odi.dwp.gov.uk/docs/res/dd/disability-survey-questions.pdf which is a long (67 pages) research report into surveying people about disabilities. I'm just skim reading it, but pages 32-34 and 36-40 might have some relevance. Thryduulf (talk: local | en.wp | en.wikt) 03:01, 9 November 2013 (UTC)

Just looking for alternatives/additions to the disability questions I've found the following. Not everything is relevant to us, I've not included questions that obviously aren't but left in ones which are less clear. It is not intended that we use all of these! Thryduulf (talk: local | en.wp | en.wikt) 15:59, 18 November 2013 (UTC)

• From the US Bureau of Labor Statistics [2] (but with adaptions for a asking a single person rather than a household by me):
  • Are you deaf or do you have serious difficulty hearing?
  • Are you blind or do you have serious difficulty seeing even when wearing glasses?
  • Because of a physical, mental, or emotional condition, do you have serious difficulty concentrating, remembering, or making decisions?
  • Do you have serious difficulty walking or climbing stairs?

  The following question isn't directly relevant, obviously, but we might be able to adapt it to be so:

  • Because of a physical, mental, or emotional condition, do you have difficulty doing errands alone such as visiting a doctor’s office or shopping?

• From various national surveys via [3]:
  • Do you have any difficulty hearing what is said in a conversation with one other person?
  • Do you have any difficulty hearing what is said in a group conversation with at least three other people?
  • Do you have any difficulty seeing ordinary newsprint, with glasses or contact lenses if usually worn?
  • Do you have any difficulty clearly seeing the face of someone across a room, with glasses or contact lenses if usually worn?
  • Do you have any difficulty speaking and being understood? [is this relevant to us?]
  • Do you have any difficulty walking 350 meters/400 yards without resting?
  • Do you have any difficulty walking up and down a flight of stairs (that is about 12 steps)?
  • Do you have any difficulty moving from one room to another? [possibly relevant for things like gallery tours?]
  • Do you have any difficulty standing for more than 20 minutes?
  • Do you have any ongoing difficulty with your ability to remember or learn?
  • Do you have difficulty using public transport? [might need to be more specific than this?]
  • Do you require the use of a wheelchair or similar device to get around?
  • Do you have difficulty in remaining seated or standing due to physical weakness, dizziness, circulatory problems, etc.? [perhaps just simplify this?]

Comparison with 2012?

How well do this year's questions correspond to last year's? (Are last year's questions and collated results available anywhere?) Are they sufficiently similar to allow for cross-comparison of survey results, and quantification of changes (and their significance)? Are there any questions that we asked last year, but are not currently in this year's draft? Thanks. Mike Peel (talk) 19:13, 26 October 2013 (UTC)

The questions and collated answers to the 2012 survey are at https://wiki.wikimedia.org.uk/wiki/File:WMUKMemSurveySummaryNov12.pdf

Particularly with the demographics questions we seem this year to be aiming to get the questions right (in terms of best practice, not offending people, etc) rather than simply duplicating what was asked last year verbatim. I haven't gone through and compared all the new questions, but it would certainly be a useful task. Off the top of my head I know that the age question this year is proposed to be to enter in your year of birth, but that will just require simple maths to correlate to the ranges we asked people to pick last year. Thryduulf (talk: local | en.wp | en.wikt) 21:30, 26 October 2013 (UTC)

Yep the idea is twofold 1) Get sections that are good enough to ask again next time to get that consistency 2) As many as possible crossover questions (some have been carried over in their entirety and the demographics questions will hopefully be better, if different, but still be able to form some sort of comparison). I'm also of a mind to do this every six months now rather than annually to hopefully get a better sense of shifts over time - it should be fine especially if we share the meta data on here and let the community pull together an analysis rather than staff doing all of that? What do you think? Personally I think the questions this time are already much better Katherine Bavage (WMUK) (talk) 12:45, 28 October 2013 (UTC)

Thanks. :-) Improvements are good, but consistency is also good. E.g. when we're asking about age brackets, it's better to stick to the same brackets than change them (unless there's a good reason to change). It's good to see what's changed since last year wherever possible (and then to speculate on why that change has taken place!). I've expanded/changed the draft demograph questions to match up with last year's questions where reasonable (although the education questions could probably do with a fresh look). Thanks. Mike Peel (talk) 22:27, 28 October 2013 (UTC)

Languages

Translation

Thinking about languages, we should (if possible) probably get this survey translated into at least Welsh and give people the option to take it in that language also. My ability in the language isn't up to the task unfortunately. Thryduulf (talk: local | en.wp | en.wikt) 22:04, 26 October 2013 (UTC)

List of languages

Should the language question also include immigrant languages? Thanks. Mike Peel (talk) 09:36, 27 October 2013 (UTC)

Possibly. We shouldn't include too many though to avoid overloading the table (there is an "other" option), perhaps the first 6 (>200,000 speakers) or 10 from Languages_of_the_United_Kingdom#Current data as of 2011 census? If we include any more than are there currently then I think we should move English to the top row and then list the others alphabetically. Thryduulf (talk: local | en.wp | en.wikt) 10:29, 27 October 2013 (UTC)

I am not sure the point in including immigrant languages - unless there is an active engagement on the Wikipedia in that language. Perhaps it would make more sense to ask about Wikipedias in which languages do people edit (see M:List_of_Wikipedias). So maybe not provide a list, but let people say what they wish. (Cornish for instance has 10 active editors, I dare say there may be more active editors in Polish, French and a number of other languages.) That way we might also get responses including Simple English, Esperanto, Latin and Anglo-Saxon. The issue should be what information we wish to gain from this question.Leutha (talk) 13:34, 27 October 2013 (UTC)

Editing may not be the same as actively contributing for the more gnomic. I have made contributions of different kinds to projects in several non-English languages without being able to read the home language, for example more than 700 edits to the Welsh Wicipedia and I don't have a clue about the language, whilst my main project, Commons, does not in theory have a home language and many contribute there without English skills quite happily. Understanding more of how this data would be used would make it easier to see how to pitch the question. --Fæ (talk) 14:27, 28 October 2013 (UTC)

Projects

It would be useful information to know which WMF projects people are engaged with, but should this go in the general (non-anonymised) or the demographics (anonymised) sections?

If there was an event that would be really useful for say a Wikisource regular then if the information is in the first section, then the Wikisource regulars in our membership could be approached about attending. However we deliberately don't require WMUK members to associate themselves with a an online identity, and for smaller projects, particularly non-English ones, there wont be many people it could be. Thoughts? Thryduulf (talk: local | en.wp | en.wikt) 10:41, 28 October 2013 (UTC)

Engaging in what sense, as editors or readers? Or both? Richard Nevell (WMUK) (talk) 10:49, 28 October 2013 (UTC)

I don't want the survey to get too massive, and so I would limit to editing? Maybe a cunning question could try and draw out levels of engagement a bit though? Katherine Bavage (WMUK) (talk) 11:25, 28 October 2013 (UTC)

(edit conflict) I was thinking as editors, but you make a good point that we should probably ask about both. Perhaps something like:

• Which Wikimedia Foundation wikis do you:

• Read regularly? [free text]
• Read occasionally? [free text]
• Edit regularly? [free text]
• Edit occasionally? [free text]

An alternate approach would be as a tickable matrix. That would get very long if we list all the languages separately, so either just do it at the project level or separate English and non-English wikis (e.g. "English Wikipedia", "Other language Wikipedia"). Thryduulf (talk: local | en.wp | en.wikt) 11:31, 28 October 2013 (UTC)

Grants

The text refers to 'microgrants' even though we have abolished the distinction between macro and micro grants now. Can we just say 'grants'?--MichaelMaggs (talk) 14:17, 28 October 2013 (UTC)

The pages at Grants don't indicate that the distinction has been abolished... Also, from the July board meeting minutes: "The differentiation between microgrants and macrogrants will remain, but all grants will be judged by the grants committee, with authority delegated to JD for him to purchase." Thanks. Mike Peel (talk) 14:27, 28 October 2013 (UTC)

My memory must be going. --MichaelMaggs (talk) 14:46, 28 October 2013 (UTC)

Length and utility

This is now getting very long, and these is a danger of adding more and more things just because the answers would 'be nice to know'. I'd suggest going through the list and looking critically at each and every question to ask "Why do we need this information? What are we actually going to do with the answer?". If the charity is in practice unable to act on a particular response, we should not be asking the question. For example, how would the charity act differently if we found we have nobody who speaks Irish Gaelic (quite possible) than if we have 2 people who speak that language? --MichaelMaggs (talk) 14:26, 28 October 2013 (UTC)

Indeed we need to do that. I've been working with the idea that we produce a large corpus of potential questions from which we select the ones we actually want to ask. Thryduulf (talk: local | en.wp | en.wikt) 14:59, 28 October 2013 (UTC)

Having reviewed the list I am happy about why I think the charity and perhaps more importantly the community could use the information in each case to improve its offer. The idea would be that the results are public and can inform lots of decisions; where we hold events, how we make ourselves available for consultations, what information we prioritise communicating to members and so on. There are some long lists of options but in fact only 25 questions - the fifth sections (Demographics) is a separate survey.

I think the potential questions to final draft is probably overkill in this content as the whole point of doing it on wiki is of course we can play about with the text :-) If you get chance perhaps make notes about which specific questions you thought were superfluous or over-long? Katherine Bavage (WMUK) (talk) 18:30, 28 October 2013 (UTC)

Oops! To answer the specifics on Irish Gaelic - I would probably follow up and ask if they were interested in contributing to relevant articles or projects :) There might be a case for reducing options here to major native minority langauages and 'Other' though? Lets be bold! Katherine Bavage (WMUK) (talk) 18:32, 28 October 2013 (UTC)

Well, perhaps the charity could look at practical steps like contacting Coláiste Feirste as regards doing some work with them. They are only a phone call away.Leutha (talk) 21:18, 28 October 2013 (UTC)

:) Good tip thank you. Yes, that's the sort of follow up that might make sense Katherine Bavage (WMUK) (talk) 11:27, 29 October 2013 (UTC)

Wikisource survey

I've just learnt of the existence of a Wikisource survey at https://wikimedia.qualtrics.com/SE/?SID=SV_6taoDVAWcg6vSIZ

Even if you don't know anything about Wikisource (I hardly know anything), its worth looking at to see how it is formatted. My overall impression is that it is very poorly accessible using a lots of different interface types, including drag and drop and sliders. The question about gender gives respondents a single-character text box to input their answer. I'm not awake enough now to usefully feedback to whoever is responsible for it, but I probably will tomorrow. Thryduulf (talk: local | en.wp | en.wikt) 00:46, 29 October 2013 (UTC)

Number of options

We now have 12 options for education and 20 for ethnicity. This really is getting a little out of hand. We could do education as 1) Up to GCSE/A level or equiv, 2) Degree level, 3) Professional, 4) None. As has been said about other questions, there is nothing that we can do with the distinction between some of these responses, e.g. between Foundation and Masters degree so why even ask? On ethnicity, why not just ask "To what ethnic group do you belong?" and somebody in the office can allocate what appears the most accurate group based on the response?

By contrast we have only three questions on disability (which is conjoined with old age), which is odd since disability and old age are two of the things most likely to prevent people from participating in our events. More so, IMHO, than some of the factors that have been debated so extensively above. It's clear that the questions in the survey are the result of what people in the community and the office feel are the important things, but if we had a couple of people in the office in wheelchairs, or living off the state pension, for instance, we might have a very different survey. Can we beware of blind-spots please. Philafrenzy (talk) 13:43, 18 November 2013 (UTC)

The number of options for ethnicity are because those are the options used by officialdom. I'm slightly horrified at the thought of assigning an ethnicity to someone based on anything other than self-reporting because I can foresee all sorts of bad happening if we get it wrong.

As for disability, yes you are right we do need more and I'm actively working on that right now. For things like ethnicity it has proved much easier to find what is regarded as best practice than it has for disability, hence that has ended up being last. Thryduulf (talk: local | en.wp | en.wikt) 14:04, 18 November 2013 (UTC)

It might be "best practice" but the result is 20 options. That does not seem very practical practice to me. I am actually proposing self-reporting in free-form. I think you are worrying unduly about something that will never happen. It is not like WMUK will be sued for misunderstanding a form is it? It's all aggregated in an anonymous form anyway as I understand it. By the time we have incorporated all the best practice into this survey we will be lucky if anyone gets to the end. Philafrenzy (talk) 14:16, 18 November 2013 (UTC)

The problem with free-form reporting is that it is not comparable and open to misinterpretation. If someone answers something like "Black" or "Polish" how do you assign them to a standardised category correctly and reliably? Say we end up with results that show we have no Bangladeshi members, but actually one or more members of Bangladeshi origin who just put "Asian" in the free-form field. What happens when they see the results and complain about racism? We don't have a leg to stand on. It may or may not be likely to happen, but it's not worth taking the risk. Alternatively we could read the results as indicating that we don't need to specifically reach out to one community, believing that we have good representation whereas actually we are very under-represented and end up furthering systematic bias. Thryduulf (talk: local | en.wp | en.wikt) 14:47, 18 November 2013 (UTC)

In terms of people finishing the survey, the number of questions is more important than the number of options (although both do have effects), but we should probably think about the order we ask questions in to gather the most useful information first. Thryduulf (talk: local | en.wp | en.wikt) 14:49, 18 November 2013 (UTC)

Well the whole idea of being in a "standardised category" is rather problematic in the first place isn't it? Who are we to assign them or put people in them? Someone might support Pakistan in cricket but regard themselves as British otherwise to use a well-worn trope. And we do in any case ask people to self-report in several of the questions, including that one, where they don't fit in any of our chosen categories. But now I am getting a sense of deja-vue about a discussion above. Isn't it a little racist in the first place to be categorising people in this way? Philafrenzy (talk) 15:32, 18 November 2013 (UTC)

The categories are the ones that the government uses for all their statistical purposes, so if we are going to use any categories using these ones is the best - making up our own arbitrary ones would potentially be racist if we got it wrong. If we don't ask these questions, we can't know which groups we are reaching and which we aren't. Thryduulf (talk: local | en.wp | en.wikt) 16:09, 18 November 2013 (UTC)

OK, we are protected from accusations of racism because we use the same arbitrary categories that the government uses is the argument. There really are no "official" categories here I am sure and we ought to have the courage to design a survey in our own way that maximises responses that we can use. Anything else is merely pandering to a confused idea about what racism really is (not that I am claiming to be an expert). I am not arguing for the questions not to be asked, that seems inevitable, I am arguing for a process that does not have 20 options. Philafrenzy (talk) 16:24, 18 November 2013 (UTC)

I'm afraid I agree with Thryduulf here. Identity is a complex issue, more complex than a survey can capture. So, with a view to doing the best we can; this is standard practice and mostly likely to capture accurate information without becoming offensive.

Sidenote: I did't find the section title particularly constructive. Can we please express our concerns/opinions without tipping into being derogatory of staff and volunteer work? Katherine Bavage (WMUK) (talk) 16:28, 18 November 2013 (UTC)

You have lost me now Katherine. I don't think anyone has been personally criticised here. Please explain. Philafrenzy (talk) 16:34, 18 November 2013 (UTC)

Ah I thought it could be that. 'Getting out of hand' could be seen to be quite negative or critical - more so perhaps than something more neutral such as 'Making questions to long' or 'Offering too many options' I can assure you the survey is in good hands - we're just taking a different approach to the one you would prefer (which seems to be very minimal) and one that is closer to ONS and other surveying organisations best practice guidelines. If it was just a turn of phrase then no worries. Katherine Bavage (WMUK) (talk) 16:41, 18 November 2013 (UTC)

I have just changed it though I remain puzzled about how it was wrong to start with. 12 options for education and 20 for ethnicity! Let's put a zero on the end of the membership numbers then a lot of this will fall into place and we might actually have some data we can work with. Philafrenzy (talk) 16:51, 18 November 2013 (UTC)

Let's put a zero on the end of the membership numbers then a lot of this will fall into place

Here here! Katherine Bavage (WMUK) (talk) 17:07, 18 November 2013 (UTC)

I like your spirit young Katherine! Philafrenzy (talk) 21:15, 18 November 2013 (UTC)

FWIW I think you (Philafrenzy) might have a point about eduction levels (I'll look again when I'm more awake), didn't have a problem with the section title, and agree with Katharine's sentiment (if not her spelling ;) ). Thryduulf (talk: local | en.wp | en.wikt) 22:25, 18 November 2013 (UTC)

I'm sorry I can't spell :-( Anyway! Hopefully we can close this today! Philafrenzy - I'm posting this on the watercooler too but we will be changing the way we report membership numbers to make it hopefully easier to identify patterns of problems/success so we can get that 10X recruitment going on :) Katherine Bavage (WMUK) (talk) 11:56, 19 November 2013 (UTC)

Education options are still too long and we need to explore disability more fully (without being too intrusive of course) mobility, sight, hearing, all major blocks to participating. I hope there will be a graph of membership stats - I like graphs. Philafrenzy (talk) 12:50, 19 November 2013 (UTC)

We've reduced the education options as much as we can to still generate information that will be actually useful for going forward. With the disability, we've (Katherine and I) decided to go for a free-form approach of asking what equipment/arrangements respondents need rather than a checklist of disabilities. It doesn't matter really to us why somebody needs a particular arrangement just that they do.

Overall, we'll be finalising the survey by the end of today for distribution to members tomorrow. Thryduulf (talk: local | en.wp | en.wikt) 13:55, 19 November 2013 (UTC)

I am sure we can do something with the education section. We have three types of degree listed. Philafrenzy (talk) 14:06, 19 November 2013 (UTC)

There is a difference between an undergraduate degree, Masters degree and Phd in terms of specialisms, research expertise and so forth. As much as between GCSEs and A-levels for instance? Katherine Bavage (WMUK) (talk) 15:30, 19 November 2013 (UTC)

Agreed, but practically it does not make much difference to us whether someone has a foundation degree or a masters. Once they are educated to roughly degree level we know they face no real educational barriers to participation (and probably below that) so we could merge those. Philafrenzy (talk) 16:21, 19 November 2013 (UTC)

This is one of the questions that doesn't seem to be very useful unless we are able to address the queries I set out above ("Why do we need this information? What are we actually going to do with the answer?"). If the charity is in practice unable to act on the response, we should not be asking the question. Are we going to focus a particular activity on members with Masters degrees, for example? --MichaelMaggs (talk) 16:33, 19 November 2013 (UTC)

I have been making this point all page long Michael, but we need some of the questions in order to start tracking changes from year to year. When the membership is (hopefully) much larger some of the finer detail will have more validity as the results will be more statistically significant because the number of respondents will be higher. This is a good opportunity to get the right questions for comparability from year to year I suppose. Philafrenzy (talk) 16:42, 19 November 2013 (UTC)

We know that people with PhD's are typically over-represented on the projects; it would be interesting to know whether that's a similar situation with members (I suspect it is). It's useful to have sufficient options to be able to compare the results with other surveys; I don't think there's too many education options here at all. Thanks. Mike Peel (talk) 19:58, 19 November 2013 (UTC)

Indeed, some of the information we will get isn't going to be immediately useful. We need it so that information we get in the future is useful, for example so we can make meaningful comparisons. My involvement with this started when I asked what we know about our members and why they are members, and the answer came back that basically we don't know much. [4] Until we know who the over- and under-represented groups are we can't meaningfully target them. Who we attract and why will also educate us about ways to increase the membership - and then how successful we have been. Thryduulf (talk: local | en.wp | en.wikt) 20:37, 19 November 2013 (UTC)

Thank you

All being well the survey should be sent to members sometime tomorrow morning. Thank you to everyone who has helped bring it together.

Please take the time to answer it. We want as many members to take part as possible so the information we gather is as relevant and representative as possible. Thryduulf (talk: local | en.wp | en.wikt) 20:46, 19 November 2013 (UTC)

I didn't receive one. Philafrenzy (talk) 18:42, 23 November 2013 (UTC)

Mine arrived just after 12:30 yesterday (Friday), which is later than I thought was the plan, but it should be with you by now. Presumably you've checked your spam folder, etc? Thryduulf (talk: local | en.wp | en.wikt) 21:52, 23 November 2013 (UTC)

Yes, nothing. Philafrenzy (talk) 23:14, 23 November 2013 (UTC)

It were sent to you around 12:30pm on 22 November. CiviCRM reports 100% Successful Deliveries. I'm not sure why you didn't get it. :( Katie Chan (WMUK) (talk) 08:25, 24 November 2013 (UTC)

Philafrenzy, it will be worth confirming with someone in the office that the email address they have for you is correct, as sending it somewhere you weren't expecting it could account for a successful delivery you haven't received. You could check your emails again just in case you overlooked it (it was sent from membership@[the WMUK domain] with the subject "Wikimedia UK 2013 members' survey" if that helps). Other than that I can't think of anything else at the moment :/ Thryduulf (talk: local | en.wp | en.wikt) 09:22, 24 November 2013 (UTC)

I can live without it, but isn't it a bit odd that we had 100% successful delivery? How many were sent? Shouldn't some of them have failed? Philafrenzy (talk) 12:58, 24 November 2013 (UTC)

Well if every member has kept their email address up to date then there is no reason why there shouldn't be 100% success rate. "Success" in this context most likely means that the WMUK server has not received any rejection messages from the servers associated with each email address. It is not a guarantee that it has reached the users inbox (at least some spam filtering is done silently for various reasons) or that they have read it. A quick google suggests that different organisations cite average response rates to email surveys of between 11% and 40% of recipients. I don't know how many were sent, but in theory it should be equal to the number of members at whatever point the email list was generated (less any we don't have email addresses for, which I imagine would be a very small number).

Even if you aren't fussed about receiving the survey it would help if you could verify your email address - if it was sent to the right place that might be indicative of a problem at the WMUK end that needs fixing. Also it means you shouldn't miss anything in the future you do want. Thryduulf (talk: local | en.wp | en.wikt) 15:26, 24 November 2013 (UTC)

Yes indeed. (Whether everything was working correctly was my principal reason for enquiring about the matter.) Philafrenzy (talk) 17:51, 24 November 2013 (UTC)

Alcohol

Far too late now of course, but it has just occurred to me that maybe we should have added events being held at venues that serve alcohol as being a possible barrier to participation? Meetups are typically held in pubs so this would be the sort of event most affected (I expect) if such is a barrier.

It'll be interesting to see if anyone surveyed specifically mentions this. Certainly if they do it'll be worth considering for inclusion on the next survey. Thryduulf (talk: local | en.wp | en.wikt) 17:51, 23 November 2013 (UTC)

This has been discussed several times at the London meetup - pubs are the default social space in British culture but they might deter Muslims and others who don't drink, or parents with children. Philafrenzy (talk) 18:41, 23 November 2013 (UTC)

Indeed, but we're unlikely to find out at meetups in pubs if anyone actually has been put off attending a meetup in a pub. If the survey finds that people are actually being put off then its worth us investing effort into finding an alternative venue, but if the off-putting is purely theoretical then it probably isn't. Thryduulf (talk: local | en.wp | en.wikt) 21:59, 23 November 2013 (UTC)

Well most members are pub meetup attendees anyway I think so are unlikely to object. We need to ask the people who aren't members which won't be happening in this survey. Philafrenzy (talk) 22:17, 23 November 2013 (UTC)

Reminder

The survey closes today I think. Is there going to be a reminder to those who have not replied? Could gather a lot who missed the original email. See discussion on the Watercooler. Philafrenzy (talk) 12:03, 6 December 2013 (UTC)

Katherine's en route to Edinburgh at the morning for this weekend's board meeting, so internet access on the train is limited but I've let her know this question has been raised.

Looking at the initial email about the survey I don't see a deadline, and the SurveyMonkey collector isn't set to close at a particular time or date. Was it agreed to close it today? I've not been involved in the logisitics surrounding the survey so don't know that detail. Richard Nevell (WMUK) (talk) 12:30, 6 December 2013 (UTC)

I saw somewhere today that this was the deadline, perhaps it is just an internal date for collating the replies. Do you know how many have replied Richard? Sorry to go on about it but given the amount of effort that went into the survey, it seems to me we should be sending a further request for a reply in a manner different from the original email (suggestions on the Water Cooler to avoid spam filters). The sample is already quite small and some of the answers will already not be statistically significant so more replies would greatly enhance the quality of results, and it is probably those who haven't replied that we particularly want to hear from. I will happily eat my hat if there has been 90% response already! Philafrenzy (talk) 13:26, 6 December 2013 (UTC)

~25%. -- Katie Chan (WMUK) (talk) 13:40, 6 December 2013 (UTC)

Well let's do something about that Katie. Isn't that half the previous one? If we take that to mean about 50 replies, we can't do much with that data. Philafrenzy (talk) 13:47, 6 December 2013 (UTC)

The 2012 survey got 54 replies, and this year's has 57 so far. I like the idea of sending a reminder out, the issue I'm faced with is how to do so effectively given 57 people already have already filled in the survey. It might be they would have to be filtered out by hand, and I'm not sure how practical that is. I'll wait for Katherine to see what her thoughts are on the matter. Richard Nevell (WMUK) (talk) 14:01, 6 December 2013 (UTC)

OK, I thought it was about 100 last time. It definitely is worth doing this by hand, we have the staff and it won't take that long I am sure. At the moment we must have used about 1 hour of human effort per reply if you include everyone who was involved. I hate to think what the financial cost is per reply! Philafrenzy (talk) 14:06, 6 December 2013 (UTC)

Sorry but no reminder. I have been ill this week and needed to focus on Wikimania and board related work. We don't actually have that staff capacity which is a point I thought I'd made clear. Please feel free to read my report to the board to get a sense of the range of my workload and why I personally can't spare a lot of time for membership work. It is something we really need volunteers to engage with.

As I said previously if you want to collate your suggestions that will make them easier to action. Also if you can make time to help more directly with delivering suggested improvements as I suggested on your talk page that would be better. We can run the survey again in six months and I'm happy to support you leading on delivering an improved process :) Next week we'll start looking at results and take these discussions forward so do put that shopping list of improvements together over the weekend if you get chance? Katherine Bavage (WMUK) (talk) 17:11, 6 December 2013 (UTC)

I realise you are very busy Katherine, anyone can do it, it doesn't have to be you, but we really ought to try to increase the number of responses or the survey will be much less useful than it could be. I am sure someone in the office can spare a couple of hours in the next week or two to do that. Running it again in six months is no good because you then have two surveys with questionable statistical validity as you can't aggregate the results. We need to increase the sample size of this one. (too few members etc. etc. cont'd p. 94) It's also likely to annoy people to run it again so soon. Really, if there aren't the staff resources to complete these common sense tasks, then there ought to be. (9 employees, budget of X, etc. etc.) I have made my point and I will leave it at that. On the other matter, I have my own business to run so there is a limit to the extent I can help. We have the staff so that everything doesn't fall on the members, but I will do what I can. Enjoy yourself in Edinburgh. Philafrenzy (talk) 21:40, 6 December 2013 (UTC)