20 Replies Latest reply on Apr 21, 2017 2:43 AM by balearicdynamics

    What's Your Ideal Way of Handling Judging and Mentoring?

    tariq.ahmad

      One of the hardest things to be asked in the community is to be asked to judge a competition.  For any given competition there might be five or six really impressive entries that are each worthy winners. You look really closely at each entry and narrow down your list as much as possible and go with the entry you believe is the strongest. Once a winner has been decided, unless there is one that clearly blows away everything else you see, you go with the candidate you believe embodies the spirit of the competition and hope you make the right decision. Often times, you second guess yourself but reassure yourself that you made a conscientious decision, and that any of the candidates you had to choose from, would be worthy winners.  At the same time, you want to reach out to the people that made your decision hard, as its their ideas or projects that made your decision hard. Having multiple prizes each month makes things easier but if this program is as successful as we hope than picking winners is going to be a very difficult job!

       

      In an ideal world, its the community members, who the community is for, who would decide.  In reality, there are a lot of practical and logistical things you have to consider if you were to go that route. Like I mentioned, you don't want to undermine the integrity of the program by having the decision based on factors other than the work itself.  At the end of the day, whatever route we go with judging would have to respect the integrity of the program.

       

      Several of you expressed concern, precisely over this. Project14 hopes to be a community member-centric program that's not about us but about you.  To ensure the integrity of the program the judging will be in part a responsibility on our end.  However, a Jury of Peers and Mentoring is a core feature of the program that makes it unique.

       

      In the comments below tell us what your ideal way handling Judging and Mentoring for our new program!

        • Re: What's Your Ideal Way of Handling Judging and Mentoring?
          balearicdynamics

          Tariq, In my opinion the unicity of Judging + mentoring is really great and it is also the way for a honest judging. Most of the times I was part of a judging team the authors was unknown. In this case if a serious judge also seriously mentor it has the opportunity to collect all the aspect of the challenge process for every challenger not only supporting him but also giving an evaluation of his job in the most correct way.

           

          Enrico

          1 of 1 people found this helpful
            • Re: What's Your Ideal Way of Handling Judging and Mentoring?
              tariq.ahmad

              My worry is that a panel of judges with community members would make this feel like a job if the same community members are volunteering.  Then it makes sense to rotate a different panel every month. What if you reach a situation where you're stuck finding volunteers in a competition that is always happening? Could advertising the judges create a bad blood in the community if people think they deserved to win but didn't?  At the same time I do like the idea of a serious judge + mentor but it seems like the best way to do this would be having their names be anonymous. I guess one route is to only allow judging by top members but then we don't want people to feel excluded as one of the core tenants of the program is to make it as inclusive for everyone.  Want to keep the focus on fun and the learning and not the prize you get for winning.   These are good ideas and I see what you thinking but tying this too closely to people's names may add a personal element which I worry would change the good mojo we have running in the community.

              1 of 1 people found this helpful
                • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                  balearicdynamics

                  Tariq,

                  as far as I know, including the previous Element14 challenges, judges are never anonymous and in this case it is even more difficult as the judges are also - at least someone of them - mentor. Also keeping the information public I think that it is not a shame to involve only the TM for first then making some reports and discussions this mechanism will set by itself.

                   

                  About involving community members the judge role is just an invitation, IMHO, like the users to participate to a challenge. A judge is a volunteer, involved in the challenge as well as the challenger. Do you know the game Eleusis, illustrated around the mid of '60s (past century ) by Martin Gardner on Scientific American? It is just our case.

                  To say this in few words:

                  • judges should mentor and the judges panel decide the winner and the challengers ranking based on the duality of the project results, and project conduction
                  • Challengers will report their feel with judges and the judges panel will become a circulating group of volunteers that have their own ranking (and can be excluded with a too low value) based on the challengers reporting.
                  • The judges are a fixed number every challenge but the subjects rotates (not necessarily all should change in two different challenges) depending on the number of volunteers judges respect the nudges panel (fixed number of persons)

                  I think that with these rules what is granted is a self-balancing system.

                   

                  Enrico

                  1 of 1 people found this helpful
                    • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                      tariq.ahmad

                      Hey balearicdynamics ,

                       

                      One idea that was discussed internally was a jury of peers and a system for signing up.

                       

                      Don't think we're ready for that as there are still some concerns.

                       

                      A concern would be that this may over complicate the process and become burdensome if the unexpected happens and there isn't a lot of interest around a particular theme, or there's a time of the year where people are occupied with family, vacation, etc. like over the winter.

                       

                      Don't want the judging to feel like work or get too bureaucratic.

                       

                      An honor system with top members, at least initially, is very sensible and should work.

                       

                      We will be reaching out further to top members for help with the judging and those that volunteered will be contacted once projects start coming in.

                      2 of 2 people found this helpful
                        • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                          balearicdynamics

                          Thank you for your details Tariq. Despite the suggested rules I mentioned (maybe the list makes things similar to a sort of "must") the idea around the Eleusis games was just to create a balanced system between judges and challengers. In a free and friendly way. I perfectly agree that the entire process by both sides (judging and challenging) should never feel like a sort of job (also a volunteering job) but should feel mostly a nice activity, a cooperation oriented to knowledge exchange.

                           

                          I have considered TMs as the starting point, then it is needed to assume the risk of the first months to experiment and calibrate the best methodology.

                          2 of 2 people found this helpful
                  • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                    pettitda

                    How about having a points system?  You could have the winner determined by a 50/50 combination of popular vote from the community and a vote by a panel of selected judges.  Then the design competition manager (or a sponsor representative) could provide the tie-breaker in case of a tie.  That could keep the integrity of the program intact and allow the whole community to have a say in who wins the competition. 

                    1 of 1 people found this helpful
                      • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                        balearicdynamics

                        As apparently your suggestion seems well balanced I think that unfortunately it is not. A panel of judges it is expected to go ahead in a politically correct direction, mentoring and judging. It is not rare that in the competitions where the community members (of any community, not necessarily Element14) arises their ranking depending on the number of friends they have, followers, contacts on other social that help him to gain votes.

                         

                        Enrico

                        1 of 1 people found this helpful
                        • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                          tariq.ahmad

                          If this could be done in a gamified where it didn't turn into a simple popularity contest that would be great.  I don't know the mechanics of how gamification and badges work but I can ask e14megan about this as this is her area of expertise.  This is different from Design Challenges s because there is no sponsor and its up to the community to choose what components they want to use.  Design Challenges are great and they're not going anywhere but we felt that a new program that wasn't tied too manufacturers or a specific challenge (a general theme) instead seemed like something that would be worthy of its own program.

                          1 of 1 people found this helpful
                            • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                              e14megan

                              Unfortunately, at it's heart, the gamification system operations much like a popularity contest (or perhaps more accurately, a participation contest). It can only track certain "tangible" actions (commenting, liking, viewing, etc.) in a basic "and/or, greater than or equal to" system. It couldn't even be used to track/award something that is "the most", i.e. "Out of these six blogs, <this one> received the most comments, likes, and views". That is something we would have to manually track and assign.

                              1 of 1 people found this helpful
                                • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                                  tariq.ahmad

                                  I get what you're saying but I'd be less concerned with people gaming the system with gamification around project entries themselves than a picture of a project in a poll.

                                   

                                  From what I've heard internally, there was something before my time that involved polls and projects. It didn't work, which is why the idea was thrown out.

                                   

                                  Of course, you can't have a perfect system of judging but you can set up a system where the focus is on the work itself and that attracts the right audience.

                                   

                                  That's why blogs containing video, images, and compelling narratives (what makes things interesting); steps to show how you did it (learning and showing what's possible), are such necessary elements for building a program that's about the work itself (think Pikea table and Ben Heck builds) and not individuals.

                                   

                                  In my experience, in the year that I've been in the community, people love seeing cool projects and are often only too happy to share stuff that they've done on their own.

                                   

                                  The most popular content as far as things that generate likes, commenting, and views are about the work itself.

                                   

                                  That's what I find intriguing about going the gamification route, or at the very least setting up badges and the like.

                                   

                                  Real human judges will always be a part of the equation of course.

                                   

                                  You'll need people on our side to ensure that the integrity of the competition is upheld and you also have a great community that does a better job of self policing than can be expected almost anywhere else.

                                   

                                  We've got lots of options.   Its intriguing to hear everyone's ideas.

                                  1 of 1 people found this helpful
                            • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                              DAB

                              Judging a competition is just like evaluating devices for a design.

                               

                              You first establish your requirements or desire outcome.

                               

                              You establish a list of attributes you want to use for comparison and rank them in importance.

                               

                              You then evaluate each device against the requirements using a fixed scale.  Sometimes we use Red, Yellow, Green, or Blue.  Red means does not meet, Yellow means partially meets, Green means meets and Blue means exceeds.

                               

                              After you compare all of the candidates you assess the scores.  If more than one candidate is all green, then you use the Blue values to decide based upon a ranking of the requirements before you begin the evaluation.

                               

                              In the end, it is usually pretty easy to figure out who did and who did not do the better job.

                               

                              DAB

                              3 of 3 people found this helpful
                                • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                                  balearicdynamics

                                  Dab,

                                  as it is also involved the mentoring - and it is an advantage IMHO - also a human factor should be considered. How the job has been done, the approach, how the challenger has shared knowledge and efforts etc.

                                  1 of 1 people found this helpful
                                    • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                                      DAB

                                      Agreed.

                                       

                                      The Mentoring issues are much more difficult to measure.

                                       

                                      It depends upon the issues encountered, the skill of the person, their receptiveness to suggestions and their willingness to work with a mentor without expecting the mentor to do all of their work.

                                       

                                      In general the same type of process is used.

                                       

                                      Set up expectations and evaluate response.  The qualitative Red, Yellow, Green, Blue system works well for this approach as you do not have any "hard" targets, just perception of accomplishment.

                                       

                                      It still boils down to identifying your expectations and deciding if they are met.

                                       

                                      DAB

                                      1 of 1 people found this helpful
                                    • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                                      tariq.ahmad

                                      This is good in theory but would all community members judge according to the same system?   Also, the community member staff that monitors these things is small so I don't know if we would have the bandwidth to follow the same system internally, month after month...  I'm intrigued by the idea though and its definitely worth exploring further. 

                                       

                                      We're still having volunteer community members + mentors but we're just trying to figure out mechanics that make more sense over what was initially proposed based on the feedback we've gotten so far.

                                      2 of 2 people found this helpful
                                    • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                                      dougw

                                      There are always risks with any judging system, but I think it would be very rare for members who are judges to purposely abuse their mandate and they would risk not being selected as judges again.

                                      The more we know about the judges the less likely they are to exhibit questionable judgement.

                                      If we know their names, already their reputations are at stake.

                                      If we see what scores each judge awards, (like Olympic diving) it is even less likely they would abuse their power,

                                      if we see their reasons for their scores, it is full transparency.

                                      On the other hand, anonymous secret ballots - where it is not known who voted, allow anyone to abuse the system without consequence.

                                      If you are really worried about it, you could eliminate the low and high judge's scores from each contestant's total. But I don't favour this as it can eliminate all the hard work of a judge.

                                      I suspect mentoring will work fine without rewards - success of mentee projects will be enough.

                                      Judge rewards, I'm not so sure, it can be a thankless job. They should at least get a special judge's badge.

                                      2 of 2 people found this helpful
                                        • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                                          tariq.ahmad

                                          Yeah, gamification and badges would definitely need to be included!

                                           

                                          I believe you could award a badge for mentoring or judging after the event and keep it anonymous while the competition is happening... 

                                           

                                          Will have to circle back to e14megan as I'm still cloudy on the mechanics, etc of how all that works.

                                           

                                          We would know internally who the judges are but as far as announcing externally, that's where there's room for debate.

                                          1 of 1 people found this helpful
                                        • Re: What's Your Ideal Way of Handling Judging and Mentoring?
                                          jlangbridge

                                          As a teacher, I already get to judge enough as it is. I know my students, I know what they are capable of, and I can "judge" them accordingly, but it isn't a nice thing to do. I hate giving out marks, since sometimes I have to give a bad mark to someone who handed in a great idea, but didn't quite get the code right, or forgot some elemental piece. What I do like on the other hand is pointing out things (constructively) that could be changed. Maybe this would be better, maybe doing this might be easier. That's the 20+ years experience talking.

                                           

                                          Judging which project is the "best" isn't too hard. Mentoring depends on specific people, and the relationship (students want to be taught, some adults don't). Marking, on the other hand, is hard...

                                          2 of 2 people found this helpful