Keeping WordCamps classy: A call for speaker reviews

32 Comments

WordCamp Los Angeles 2013

You’ve heard Ron Burgundy tell San Diego to stay classy. Fortunately, it’s not something we have to do at WordCamps most of the time, because everyone naturally keeps WordCamps classy.

But that’s not always the case.

WordCamp Los Angeles 2013

I just came back from WordCamp Los Angeles 2013, which was incredible. Amazing. Awesome. Spectacular. The first time organizers did an incredible job and it was so smooth and well put together that you wouldn’t have known it was their first time.

In a case or two, my friends and I (see the photo above) sat back listening to a speaker who struggled to “get it.” By get it, I simply mean, they misunderstood what makes WordCamps so incredible.

It’s not the speaker. It’s not the topics. It’s not the food. It’s not the location.

What makes WordCamps so incredible is how much “giving back” happens.

It’s the thread that holds everything from speakers and talks to genius bars to after parties together.

You see people give back when you see out of town guests fly in from far away (on their own dime) to share insights with the local community, or when you see locals share tips on what they’ve been learning. You can see it when people answer questions for hours – without renumeration. And you see it when people burn hours prepping a party, posting signs and handing out parking vouchers, all so others can have a great time.

These are volunteer events. For the community. By the community.

Speaker Guidelines

So as my friends and I sat back and listened to a presenter or two share more about themselves than their material, it got me wondering. How can we better protect WordCamps from this?

Already, we have some wonderful guidelines for organizers about selecting speakers.

  • Keep a mix of local and outside talent
  • Make sure everyone embraces GPL
  • Make sure they know their stuff

These are helpful and a great start. If you look on most speaker submission pages, like the one for the upcoming Orlando camp, you also see more text that gives these guidelines.

  • Your talk needs to be connected to the community. It can be front end, back end, blogging, freelance related, but it needs to be for the community.
  • You have to know how to spell WordPress (notice the capital P).
  • You have to be prepared (like have your slides done in advance).

Again, great foundations. But my favorite one is the one I’m most focused on today. Here’s the text most of us use on WordCamp pages:

  • Your session is not primarily to promote your own theme, plugin or business. If your session is to detail how you used WordPress code to build some really great aspect of your theme, plugin or business, and you’d like to present that as a sort of case study, go ahead and submit your session, but know that we’re not going to allow for a sales pitch at any point.

If you’re pitching, you’re missing the point of what the event is all about.

Speaker Ratings

I’ll admit that my first thought was that we should have a blacklist. If you get on it, you have to work hard to get off it. But more importantly, a central black list would protect all our events to make sure people who were pitching themselves would be ousted, once and for all.

After all, I once had someone ask me how I got on the “WordCamp Speaking Circuit,” as if something like that existed. I explained every event was local. There was no circuit. And while I’ve spoken at 9 events this year, the first two years I spoke at WordCamps, I only applied to those that were within 100 miles of my home.

When I explained that speaking was predominantly a localized event and that being invited other spots might take a few years to materialize, if at all, I heard the normal, “that’s too long and too much work.” And that seems like a good enough bar to keep WordCamps classy.

But in a call with Andrea Middleton, she quickly highlighted all the ways that a national blacklist wouldn’t be a great idea. She’s one smart cookie.

The more we spoke, the more I came around to the idea of speaker ratings. It’s a natural feedback loop, which is how a lot of what we do works.

A Call for Speaker Reviews

speaker-review

If we had a centralized place to capture speaker reviews for every session, at every camp, we could easily close the loop on speaker performances so that people who spend too much time talking about themselves will be noticed.

If we rate speakers on the overall experience, on their content, and their delivery, we can provide feedback that helps speakers get better – by going deeper or learning to deliver more effectively.

If we highlight was we loved, as well as what left us frustrated or confused us, we can help people adjust their topics and cover them more effectively.

Of course, Reaktiv Studios just released a plugin for ratings, so it might not be much work at all. But it would mean some work to get every camp and talk loaded into a system so that we could collect the feedback.

What do you think? You up for it?

Photo Credit to Adam Silver | Silver Lining Photography

About This Site

This site is hosted by Pagely - running on Amazon's hardware, making it lightning fast. They're a managed WordPress host which means they take care of my site so I don't have to. Starting at $24/month, this may be the perfect solution for you too.
  • http://philoveracity.com Verious Smith III

    This is a brilliant Idea and a measurable solution that would benefit the WordPress community both ways. Do you envision this located on the actual WordCamp website? Wherever it is, i think it would only increase the quality of the community.

    I think i will begin to implement this on a small scale for The Inland Empire Meetup Group.

  • http://zeek.com Steve Zehngut

    While it doesn’t happen often, “bad” speakers do slip through the cracks. Speaker ratings are a fantastic idea. In addition to helping filter out the bad, this would give all speakers the opportunity to find out from the audience what works and what doesn’t.

  • http://tri.be Shane

    I would suggest they simply add it to people’s .org profile.

    • http://shaynesanderson.com shaynesanderson

      Oh snap!

    • http://www.wpbeginner.com/ Syed Balkhi

      I second this…

  • http://twitter.com/kitchensinkwp KitchensinkWP (@kitchensinkwp)

    GREAT post! I love the WP Community & WordCamps for the basic premise that no one is paid. I do what I do for the simple premise to share, learn, and grow as a person. Along the way I’ve met some amazing people and have new friends. Simple! -Adam

  • http://marktimemedia.com Michelle Schulp

    We do have our audience review and comment on all speakers, but only organizers see them. It would be nice to be able to share that (anonymously) with the speakers and the community at large.

    • davidlaietta

      Last year we used the Polldaddy form to collect feedback from attendees, then I went through and separated by speaker, and shared information with each speaker individually (so that they only saw their feedback and not for other speakers). This had generally good feedback from the speakers, as I was told that this isn’t done that often. I’ll probably give it a shot again this year, though it’ll take a lot longer to compile.

  • http://UberQuality.com/ Jim Krenz

    As a first time WordCamp speaker (WordCamp LA 2003), I would be open to such a ratings system.

    I do have a concern: how will you make allowances for the differences in each reviewer’s background, experience and perspective? Some attendees are new to WordPress and/or WordCamp. Some people are technical and others are looking for ideas or business advice.

    For example, at WordCamp LA 2013, a very knowledgeable speaker gave a very dry, but detailed presentation. Another was much livelier (in terms of presentational style), but lacked technical depth. Another made several technical mistakes but made light of them and continued on and they finished well.

    Did I get value from all of them? Absolutely. Would I attend their presentations again? Yes. But under a strict review system, would they be blacklisted? If style, depth and mistakes were being assessed, then we risk losing some talented speakers. In my opinion, this would be a bad thing for our community.

    Thoughts?

  • davidlaietta

    Thanks for the shout out to Orlando Chris. We do care greatly about our community, and to the quality of talks that we bring in. While the criteria are multiple (most of which you’ve covered here), there are admittedly some that may not be the “best” way to do things, but perhaps the most practical in a sense.

    For instance, we’ve received a great number of responses of local speakers, which is fantastic and is one of our priorities. However, we’ve admittedly not seen all of these speakers in action anywhere before. That’s not to say that they aren’t good, and while I welcome everyone to come and present at the meetups, I don’t specifically draft people for “tryouts” as has been done before. Considering that these are people that I’ve never seen involved in the local community, I can’t offer up any good advice to the speaker selection committee on their style.

    Another generally unspoken issue is that part of the search is for “names”: people that can draw a crowd. It’s a vicious cycle almost, where someone can gain notoriety in the community, and be selected to continue speaking both for this ability to attract attention, and as they have demonstrated that they can manage a room well. These are a few of the selection issues that we face.

    We are trying to make balance, and I know that no decision will be perfect. I can state here that we’ve received about twice as many submissions as we have spots available. This is excellent, and it’s amazing to see how much of our local community and friends from afar want to attend and be involved. This is also negative in that we have to turn away as many people as we select, and it will necessarily indicate that some people garnered more interest than others. Whatever decision is made will not please all, but is made with the best of intentions.

  • http://benmay.org Ben May

    I don’t think it needs to be public (the reviews) but more a tool for organisers, and indeed speakers who are new to the game, to take the advice and try improve ( not talking about people who are flogging stuff – but just general n00bs, like me ;)

    But the feedback idea, and centrally is a good idea.

    Also gives more benefits to having a WordCamp and not a break away, independent one.

  • http://twitter.com/leehblue Lee Blue (@leehblue)

    As a speaker at several WordCamps, I would LOVE to have received more feedback. Basically the only feedback I got was searching Twitter for my name after the talk. I suspect people would be more candid over Twitter than to my face, and an official channel for receiving feedback would be great. Along the lines of what Ben May just said, public “reviews” might not be necessary because reviews suggest summarizing the talk, providing analysis, etc. And we don’t want to embarrass people publicly who are legitimately trying to do their best. A feedback channel though would be great. I suspect most folks giving a talk just want to know if they did OK and how they can do better next time. Should that be public? Maybe… maybe not… but getting SOMETHING back to the speaker would be great – I know I’d appreciate that.

  • http://joshbroton.com Josh Broton

    Sometimes I think we’re a little too nice. The lack of speaker reviews is painful. Not because I can’t give feedback, but because I can’t get feedback. Twitter mentions and personal conversations are fine, but because we value community so much, no one’s going to publicly point out places you could improve like they would in speaker reviews.

    There are open-source, community-developed options for speaker reviews (joind.in as an example), and it would be invaluable to speakers and organizers to have that feedback.

    • davidlaietta

      We do a survey after the event, and a few other camps that I’ve been to have done the same. I then compile the info in it specific to each speaker, and send it over to them.

      Anonymous PollDaddy polls let people be more honest with their critiques :)

  • http://coreyfreeman.me Corey Freeman

    I’ve only spoken once but I’d be up for feedback. I’m concerned I already know what people would say, though. “You talk too fast and stop slouching!”

  • http://www.bobwponline.com BobWP

    I think it’s a good idea if it is considered a tool, and not the sole decision for using a speaker or not. A couple reasons…

    Reviews, in any form or on any site can get out of hand. Myself I’m not a huge fan of them. Depending how they are moderated people can give bad reviews out of spite or just to be mean. And often those don’t reflect on the real value of what or who is being reviewed. On the other hand, I know people who have in the past “lobbied” to get others that they know to come in and give them a “good” review.

    Secondly, a story (I know you like stories Chris). At a WordCamp in the past I found someone that was going to be speaking sitting off my himself. He was incredibly nervous. We talked for a bit and then he went and gave his presentation. Afterwards he wasn’t real sure of himself as he said he probably made several mistakes. I found someone I knew leaving his presentation, a tech teacher at a local university, and asked her casually how it went. She said, “I could tell he was nervous, but man, his information was awesome.” I shared that with him and it probably made his day. He went on to do more presentations and eventually was hired on at Automattic.

    The point in the last one being if there was one good review like that, but several others that just focused on his nervousness, what impression would that give us? One of the things I love about WordCamps is the openness of the community and how forgiving they can be.

    I’m not saying we should have crappy speakers, but we also need to be careful if reviews are started. They can be helpful, but as WordCamp organizers, we also would need to read between the lines. And I agree with what someone said, they shouldn’t be public.

  • http://twitter.com/dimensionmedia David Bisset (@dimensionmedia)

    As a WordCamp organizer, I have mixed feelings. But we are talking about a few different things here (at least in the comments).

    It’s always been a good idea for organizers to survey attendees after the event, and to share those findings individually with the speakers. No brainer there.

    I’m not sold on posting PUBLIC reviews of WordCamp speakers though (Although i realize that “rate the speaker” sites already exist). I think this actually might imitate new or shy speakers from submitting presentations or taking part. I know of a few speakers like that – and we’ve had talks about public reviews of their presentations – and honestly the feedback i got was they wouldn’t be open to it and might prevent them from submitting to speak in the first place.

    This is one of those things that it depends HOW it’s done. I wouldn’t want a rating system like with WordPress plugins (1 to 5 stars)… perhaps something more positive (or less negative) as simply leaving comments. But again, i’m not sure if that solves my problem with it.

    Let’s go back and think about why this idea is coming up in the first place, at least according to Chris. It sounds like it’s focused on the self-promotion so I’ll run with that.

    Many WordCamps, like Orlando, have similar speaker guidelines before they submit. That’s good, but read the one’s for the past WordCamp Miami (http://2013.miami.wordcamp.org/speakers/speaker-submissions/). Notice the last bullet:

    “Be prepared to submit speaker slides PRIOR to the event. No last minute talk preparations (unless of course you were notified last minute or some other sort of emergency). First time speakers might need to submit slides for a general review.”

    I think the best way to prevent promotion-based talks is to review the slides of the presenter prior to their presenting (that’s a lot of p words, yeah). I cover this quite well in my blog post:

    http://www.davidbisset.com/speaker-slide-showdown/

    Those who “don’t get it” as Chris describes it and want to self-promote should be made to understand and “get it”. I think this is a better approach than giving them a poor public review.

    Who doesn’t it? It’s usually first time speakers (either they haven’t spoken before OR they haven’t spoken at a WordCamp before). If you are organizing a WordCamp, you should be very meticulous in the approval process if they fall into this category. If the “new” speaker is local, have them speak at a local WordPress meetup FIRST. This is an audition of sorts.

    Either way, reviewing slides and communicating with the speaker prior to the event works out for everyone – the organizers get that confidence (instead of letting them through and getting poor reviews afterwards), and speakers who “don’t get it” hopefully get an education. Worse case scenario – if the speaker is “sly” and does self-promotion (going off script from what they told you or by the slides) then you have the ability to state “we have a review process, and that speaker deliberately didn’t follow our procedures and won’t be asked to speak again”… or something along those lines.

    Sorry to get off track on slide and speaker reviews (that’s what my blog post was for), but if the problem is self-promotion I think this is a more effective thing.

    Sidenote: I’m open to having a directory of profiles that showcase a speaker’s given presentations at WordCamps. Say, if i can easily see a potential speaker has spoken at three previous WordCamps – then I can assume on some level this person is experienced and consider that in my speaker approval process.

    • http://chrislema.com chrislema

      David,
      I totally hear you – and I hadn’t landed on private or public ratings because my issue was really the self-promotion stuff. And you’re right that getting a deck in advance is very helpful.

      It’s precisely the last part of your comment (the assumption that if they’ve spoken elsewhere they’re ok) that I’m trying to head off. Because that isn’t always the case.

      This isn’t about writing negative reviews because you didn’t like someone’s talk. It’s about catching folks who are preying on our own community and protecting organizers from giving them a place on stage. My sense is that they need a centralized set of data that they can review.

      I’m open to better ways.

      • http://www.bobwponline.com BobWP

        Hey Chris, having been in the role or organizing a WordCamp, and I know you and David are as well, it seems as if it has been pretty easy to weed out those folks preying on the community during the selection process. I remember for me, it was a matter of red flags popping up in their submission, a little extra time visiting their site and social media profiles that gave me a good impression of the person and their intentions.

      • http://twitter.com/dimensionmedia David Bisset (@dimensionmedia)

        Chris,

        If we are still focused on the self-promotion thing, the majority of people that “don’t get it” (again, your words) are new speakers. At least that’s been my experience in the past 4 years. While i did focus on them in my comment, I didn’t say it was the sole source. So I agree we shouldn’t make such assumptions. And by your words, it sounds like the people who you saw at the LA WordCamp (correct me if that wasn’t the location where you saw the “bad” speakers) were “preying” on the community? So repeat offenders?

        I think, again, connecting with ALL speakers (new or not) and taking a look at their slides is a great policy. And it should be applied to all speakers regardless of their background.

        If you have seen presentations with promotion, i’m willing to bet that the organizer at the event didn’t do a review or talk with the speaker one-on-one. I think this is a problem among WordCamps in general – there needs to be a firm policy and every WordCamp needs to review and have time to talk to the speakers before and after they are approved.

        I’m not gun-ho for a central review system for WordCamp speakers at this time – private or public. I think private would be WORSE because you have potential speakers hearing about this list and wondering if that was the reason they were rejected from speaking at a WordCamp. In a way, this could be used as a blacklist (or seemingly so from the outside). I was hoping you would have shared what Andrea pointed out why blacklists are bad, because i think if one isn’t careful reviews can be used in a similar fashion.

        I’m certainly open to discussing this further – with some ideas. There’s alot of ways to do this and what I have pictured in my head might be a worse-case scenario.

        BTW, I know adding WordCamp history to .org profiles has been in talks for a while. I’ve spoken to a few already and the thinking is that eventually we’ll get there.

  • Pingback: WordCamp Speaker Blacklist Is A Bad Idea

  • http://cloudillumination.com Marviene Vulpes Fulton

    Yes. It is needed. I loved the weekend but yes, there was one speaker who I walked out on. He wasn’t teaching. No one got anything from his talk, not new business owners, not SEO, not web designers, and certainly not developers. It was hard. It feels bad to get up in the middle of a lecture and leave but ultimately I’m there to better myself and sometimes being too polite is a detriment to my goals. I do not think he should be allowed to speak again for years. After coming to future WCs he will hopefully learn and come up with a great lecture that meets the needs of at least a part of his audience.

    • http://twitter.com/dimensionmedia David Bisset (@dimensionmedia)

      I would convey this to the organizers, Marviene, once they send out a survey. If you don’t see a survey after a week or two, i would reach out to them directly (I’m sure myself or someone can do that for you – i do recaps of WordCamps on the wordcamp.org site and I hope to be in contact with them soon).

      I’m sure organizers wouldn’t have put them on if they knew, so they should be informed of this.

  • Pingback: WPTavern: WordCamp Speaker Blacklist Is A Bad Idea | A2Z Web Design Tutorial

  • http://gravatar.com/vidluther Vid Luther

    I like the idea of speaker reviews, but I would like to point out that these review systems aren’t unique to the WP ecosystem, and we should perhaps not try to build another plugin or theme, or .org site that handles this. I would instead post these events on sites/services like http://joind.in/ where anyone can do the reviews.

    The code for the entire project is open source, it’s written and used by a lot of the people in the PHP community, and may also help attract more local attendees and speakers. As much as I love the WP ecosystem and community, I would hate to see a reinvention of the wheel.

  • http://illuminea.com Miriam Schwab

    I organized the last three WordCamps in Jerusalem, and we sent out surveys after each conference to get participant feedback both about the conference in general, and about the sessions.

    The goal isn’t to identify bad speakers, but to see how we can improve for the next year. I’m not sure it’s possible to reach a situation where 100% of the speakers are great, and I think that’s ok. Especially since it’s a community event, first-time speakers are often testing the waters in a way during their talks. In my experience, the vast majority of speakers have good intentions, and with a bit of constructive feedback, they can become better.

    • http://gravatar.com/vidluther Vid Luther

      +1 on the “goal isn’t to identify bad speakers, but to see how we can improve”

      • http://UberQuality.com/ Jim Krenz

        +1 from me as well—this is the best way to protect our community.

  • weston deboer

    We all see advertising daily in our lives and have learned how to block it out. For every talk that is a pitch, I still learn something. I don’t remember there product nor who they were nor what they were even pitching. But every talk will say something that speaks to me.

    I think a review process is not a very good idea, some of the best talks were thought up the night before and or the worst speakers said the best things.

    When is WordPress becoming all high and mighty. WordCamp’s are about meeting people and interacting with like minded individuals. There will always be pitches and advertising.

    Wait did you just pitch us at the end of this post?

    • http://chrislema.com chrislema

      No, I didn’t pitch you. And most certainly didn’t pitch you on ME.

      Yes it’s true we learn to block it out. But not everyone does. After all people still give their account numbers to email senders telling them they’re going to deposit money in their accounts.

      My point wasn’t about high or mighty. It was about protecting our community.

  • http://twitter.com/cliffseal Cliff Seal (@cliffseal)

    Duly noted that I’m not a WC organizer, but I tend to agree with David: requiring slides in advance and actually reviewing them is crucial. When I speak at non-WordCamp events, I often have to submit weeks in advance—it makes me more prepared by default, and gives the organizers a chance to request any changes. In fact, my first WordCamp speaking experience was a bit marred by a lack of follow-through on this: I submitted my slides in advance as requested, but they were never reviewed. I found out when speaking to Jen Mylo afterwards that I’d ignorantly included some non-kosher items (related to the GPL). She was gracious and all was well, but it could’ve easily been avoided if someone had scanned my presentation and let me know that some items should be removed.

    I also think getting out in front of newer speakers would be helpful. I’ve posted my willingness to help on /community: http://make.wordpress.org/community/2013/09/14/making-better-speakers/

    In terms of reviews, it’s got its pros and cons, but the detailed feedback I’ve gotten from non-WC speaking engagements have been incredibly useful. I was shocked by how many attendees were willing to rate my presentation on 10+ metrics and give a brief review. The folks at Web Conference at Penn State are pros at this.

    Thanks for using your sway to get some movement on this, Chris!

  • http://eliw.com/ Eli White

    I’ll second (third) the suggestion to just use joind.in, that was the genisis of that project in the PHP community, for rating speakers in a centralized area.

    If WordCamps use joind.in, then not only will you have your centralized rating system, but you’ll be able to benefit from getting ratings of speakers who haven’t ever been at a WordCamp before, as well as vice versa, sharing back to the ‘conference speaker’ community and help out other conferences by letting them know how good of a speaker someone who formerly only spoke at WordCamps, but was now reaching out to related communities.