48 Replies Latest reply on Jun 28, 2019 12:58 PM by starbasessd

    2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review

    rscasny

      Recently, I selected roadtest applicants who were both new roadtesters  and have not participated much on the community, primarily because I have had some other members tell me that I am using experienced roadtesters too much. (This is never my intent; I select the best applicants. When I roadtest writes only a 1-sentence application, I don't have much to go on and these folks usually don't get selected.)

       

      I had to report roadtest results to 2 sponsors. Well, the results for these 2 roadtests were not great: 4 roadtesters haven't done their reviews, they haven't logged in for 3 to 4 months, and they have not responded to all our follow ups that we do for every roadtest. Before I selected these 4, I contacted them and they agreed to commit to delivering the review in 60 days.

       

      Now, 2 roadtests is a fraction of all roadtests. For many roadtests we get 100% compliance. I'd say overall we are getting a compliance rate in the upper80s/lower90s, which is good and a far cry from where we were 2 years ago when we were at the sub-20% level.

       

      Speaking for the sponsor and myself, we understand that in some situations people will be unable to complete the review. After all, people do get ill and emergencies do arise in everyone's lives, so I never expect to have 100% compliance for every roadtest. I recall someone who had a tree fall on his house in an ice storm. That's a major tragedy in my book. Of course I would not expect a review. But these instances are atypical and very few compared to all roadtests and roadtesters I deal with.

       

      While we give the official roadtesters the kits and the shipping for free, someone does pay for these things. Typically, the sponsor pays for the kits and element14 pays for the shipping costs to the roadtesters. But we are more than happy to do this because we value our roadtesters' opinions and we are happy to absorb the costs when the reviews are published.

       

      But the question comes to mind: what happens when a roadtester doesn't complete the review:

       

      1. The sponsor does not receive all the feedback he is seeking when he enrolled in the roadtest program as a sponsor.

      2. Someone who does not complete the review prevents someone who could have done the review from doing it.

      3. It prevents our members from learning about the product.

      4. It damages the credibility of the program.

      5. The delinquent roadtesters prevent themselves for being considered for a high-priced product. If I can't reliably use someone for a $35 US roadtest, how can I expect them to do the review for a $3500 US roadtest!

      6. I am placed in the embarrassing situation to explain why 4 people I selected did not do their reviews.

       

      There are other reasons but 6 reasons is enough to make my point.

       

      It is my responsibility to get the required results for both our sponsors. My bosses also have expectations and expect that I will solve problems. I want people to use the RoadTest program to have fun with technology,  and get some products or equipment that they may not have the budget to get on their own. But I also need mature, responsible members who are engaged in our community, demonstrate an appropriate level of professionalism, and are responsive to our follow up requests. I like to think most roadtesters are professional. I surely do not think these 4 delinquent roadtesters represent the RoadTest group.

       

      But going forward, I will have to modify my approach in selection somewhat. And I probably will not hand out all the kits if I don't feel I have enough roadtesters who are knowledgeable about the product, and are mature, professional and reliable. If you were in doubt that I am very serious about people completing their reviews, I hope this discussion makes my position very clear. I apologize to all my loyal and responsbile roadtesters who do complete their reviews. But this is a community and I have to communicate my concerns as well as be open to new ideas and feedback: it's a two-way street.

       

      If you can't finish the review, contact me and tell me or my colleague danzima why. If you need a bit more time, that's fine if you need to do more testing. (Needing more time to finish your Differential Equations homework or finishing your digital electronics lab is not a reason for needing more time.) Communicate and keep us in the loop; you will find in your career that developing communication skills is as important as honing those engineering skills. Even if you received a non-working product, you are still required to post a review. Put it out to the members: "I took these readings and I believe it's faulty. Help me troubleshoot it." There are plenty of eager members who are more than happy to help.

       

      I want everyone to benefit from a Roadtest. I think completed roadtests are a great place to learn. I think they are a fantastic vehicle for the new engineer who is trying to shine in their first job interview. Read our e-book on IoT careers to learn more shining in a job interview: https://www.element14.com/community/community/publications/ebooks?ICID=hp-iotcareerebook-rotational#?ICID=ebooks-downloa…

       

      But you can't reap the benefits of the roadtest program if you don't do your reviews.

       

      Sincerely,

       

      Randall Scasny

      RoadTest Program Manager

        • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
          Fred27

          Unfortunately you are going to get some people like that. I don't envy your job having to pick road testers. It must be tricky trying to balance the requirements of the suppliers (who are the ones that count, really) but also trying to keep the community on side. I can tell you how I'd approach it, but obviously I'm not aware of other constraints and restrictions you're under.

           

          I'd let people prove themselves on a simpler, cheaper item before letting them lose on anything bigger. Let them know that too. If a shiny desirable piece of kit has "Must have completed 2 good roadtests" as criteria maybe that might motivate people to put some effort into the smaller ones. Whilst I can understand how some people might think some members get to do all the good stuff, it's the suppliers who are paying for this to get done. In my opinions we road testers get the better end of the deal. Gough Lui is a great example. [I hope you don't mind me mentioning you by name, Gough.] He may roadtest loads of cool things and be regularly picked, but you can't argue with the results. He comes through with a great road test every time. A few other familiar names fall into that category too. I'd certainly pick someone like that over an unproven new member for the bigger stuff.

           

          I also feel that saying "free" too much is a bad thing. I see it a lot if the road tests are promoted on twitter for instance. I suspect that's a good way to get more applicants, but are they the ones you want? Are they the ones who will complete them. Sure, the small print says you need to do the roadtest, but the freeloader types are probably still hearing "free". (Extra bonus - less applications to trawl though. That Azure Sphere one will certainly take you a while!) As far as I'm concerned it's a trade - some hardware for some time and effort. Maybe it legally has to remain an informal deal, or it has to technically be free, but it's a agreement as far as I'm concerned. (I tend to get my road tests done quickly. I realise I feel I'm in debt until it's done. It's like a friend has lent you some cash when you forgot your wallet - and you want to make sure you pay them back.)

           

          Anyway, the road test program is a great asset for E14 and improving all the time. We all appreciate the great job you're doing. I hope it continues and continues to improve.

          4 of 4 people found this helpful
            • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
              lui_gough

              No problems - happy to be mentioned Fred27, especially when it's positive in nature . Thanks for your comments - it's great to be acknowledged for my efforts. I am truly humbled, but I'd have to say that on the whole, practically all of the Top Members have a track record of delivering quality RoadTests, as do a few new (but active) members. Some RoadTests are more difficult than others and life can get in the way (it certainly did for me in my triple-concurrent RoadTest recently) but I don't take my responsibilities lightly. There are, of course, the jealous types and I don't doubt that the statistics may look lopsided - but RoadTests are not a game of "chance", so I don't take those comments to heart. Participate or not, this is ultimately your own choice - win or not, this is ultimately the choice of element14 and the sponsor.

               

              Needless to say, I do try to make sure I deliver what I promise, or if not, there's a good reason for it and perhaps some bonus content to go with it to make up for it. My reviews can be a bit dry and systematic - but a good quality on-time RoadTest ensures that everyone is happy. This means that element14 forum members and other visitors get some good content to look at, it means the sponsor is happy and gets some exposure/engagement/conversation about a given product (positive or negative) or some feedback to help fix issues, it means the element14 community raises its profile and perhaps means more RoadTest opportunities for all in the future. In my opinion - fail to deliver, and it could cost more than just *your* ability to get future RoadTests regardless of the value of the item.

               

              We've mentioned the whole issue about "free" in the past and whether it was attracting the right types of applicants. At one stage, RoadTest "by invitation" was an alternative where members would be selected for a given product instead, which would have been completely opposite. I suppose, the issue boils down to fairness (perceived or actual) and barriers to entry - so maybe we just have to accept that in order to give new members a try, this is the price to pay. I wouldn't be against having tiered eligibility depending on items - but whether this is realistic from an administration point of view and perhaps block potentially good RoadTesters from going for an item is something to think about. We all have our own areas of expertise - the lower cost items are almost always EVMs/dev boards, so perhaps some people may not be able to deliver a good RoadTest for those and end up unfairly excluded for something else more expensive that is perhaps in their area of expertise. I'd love to see us discover more like-minded highly-technical engineers, hobbyists, etc and would love to see reviews from them.

               

              I know rscasny has a tough job balancing it all, so I'm glad that we do get some insight into the challenges that are being faced. I'm not always easy to deal with either - I often come up with queries and special requests to contact manufacturers during RoadTests, and I've left some honest but not-as-favourable reviews in the past, so I'm glad to still be welcome around here. Above all, I wouldn't sleep comfortably if I wasn't being honest with myself, the community and the world.

               

              - Gough

              4 of 4 people found this helpful
              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                mom2673

                I could not agree with you more on the Word FREE enticing reviewers who arw only there because the item is FREE. Unfortunately, it doesn't leave a lot of reassurance and trust for those of us who are new to this website and would be grateful for the chance to test out a product. Maybe you could have the new users (speaking as one myself) purchase the product then, offer a refund once their review has been completed.  I would be very willing to do that since chances are it's a product I probably wanted to buy anyway.

                 

                Well, I'm just glad I stumbled across this site I'm search of information for an IT project I'm going to be undertaking soon.  You have NOT heard that last of me!

                Thanks all for a really great, informational site. I'm looking forward to learning lots from the questions I will surely have, lol.

                3 of 3 people found this helpful
              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                shabaz

                It's astonishing yet tiresome dealing with unreliable+irresponsible/unprofessional people. Sorry you have to experience it : (

                Going through the review process helps in learning stuff, building a portfolio of experience, and improves communication skills no end, as we've probably all observed in each other.

                • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                  DAB

                  Sadly, that is the outcome I predicted several years ago.

                  If you want to satisfy the vendors, you have to use trusted road testers.

                  You can always allocate a couple of inexpensive gear tests to newbies so that they can prove they belong on the trusted list, but you cannot risk expensive gear to newbies.

                   

                  DAB

                  1 of 1 people found this helpful
                  • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                    colporteur

                    No doubt there are some individuals who want a chance to get their hands on free shite free stuff. Professional members that frequent this site realize there is no free. You may note I am one of the first people to jump on messages that are looking for answers without providing sufficient details. The free goods whether they are product or answers are an exchange for an investment commitment. The old adage your word is your bond doesn't always apply. I appreciate things come up that may inhibit or delay completing a RoadTest. That is when you are "required", I used the word specifically, to send a note outlining your failings and how you will correct the situation. I know I don't like writing those letters!

                     

                    I do hope the process can retain some mechanism for proving new entrants. New people are required to evolve and expand the content. I write for pleasure and have the bonus of getting paid for some of my scribbles. I have a writing style that has change little over the years. I have a formula that works but I am cognitive of the understanding some change is necessary to stay fresh and expand your audience. I try to incorporate change to improve the product, I find it hard but a must. New participants are needed have to evolution. Giving up is not an solution.

                     

                    I'm sitting here thinking, maybe an essay on why you would be a good candidate might be a solution. Then I realize JHC<talking the judeo-christian gods name in vain if translated>, someone has to read the dribble and that someone is you.

                     

                    I worked in a Junior hockey program that gave the university eligible players $1000 if they submit an essay on how hockey impacted their lives. One of twenty four players on the team could win the free money. Only half of the players were in university so that improved the odds. No one, I mean no one, submitted an essay before the deadline. I know if I was playing hockey back when christ was a cowboy, I could have used the $1K.

                     

                    I ask you to find some mechanism to enable new participation but I accept the process has to be efficient in order to be effective.

                     

                    Sean

                    2 of 2 people found this helpful
                    • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                      stevesmythe

                      I hope that the two roadtests were for relatively inexpensive products.

                       

                      Sadly, you are bound to get people who see their opportunity to get something for nothing but, as you have said before, if you don't take some sort of risk, you'll never get new roadtesters. This time last year, you took a chance on me as I had not come across the element14 community before and I am glad you did. I delivered that first roadtest and have done several others since then. The benefits for me have been way more than the "free" products - I have learned so much, not just from the stuff I have tested, but from reading the reviews that others have written, and people's blogs and design challenges. It is a real community, albeit virtual.

                       

                      I'm guessing that you already tend to go for "tried and tested" roadtesters for the expensive gear, which is what other people are suggesting. I see it as an apprenticeship in a way - build up your skills, experience and reputation before taking on the bigger challenges.

                      1 of 1 people found this helpful
                      • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                        genebren

                        Randall,

                         

                        I understand the desire to have new blood involved in Roadtests, but maybe it is should be necessary for Roadtesters to have participated in the community first, by either competing in other contests or blogging prior to being approved to participating in a Roadtest.  But not matter where you set these thresholds for participation, some people will game the system and renege on their promise to complete the necessary review.  It could be the allure of something for nothing, or some other personality flaw, that would allow a person to do this.  I just don't know, but you do have difficult choices to make and from where I stand, I believe that you have done the best job possible of fairly assigning these opportunities to wide section of this community.

                         

                        Thanks for all that you do!

                         

                        Gene

                        3 of 3 people found this helpful
                          • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                            dougw

                            I kind of like the idea that a new member could or should publish a road test/blog about something they already have to establish some credibility. This avenue is already available of course, but it isn't mandatory. If it were required for some road tests that a previous road test had to be published, it wouldn't be that big of a barrier for new members, and it could be a nice boost to site content. There is usually plenty of time between a road test announcement and the application deadline to publish a road test. If this criterion was trialed on half the road tests, it would be easy to see if it worked better than the current requirements in terms of number of applicants (deterrent factor) and successful completion of road tests.

                            How many currently successful road testers would have been keen enough to publish an un-sponsored road test prior to applying for a sponsored road test?

                            It is a tricky question - obviously for the few members who already publish un-sponsored road tests it is a no-brainer, and with what I know now, I wouldn't hesitate to publish an un-sponsored road test, but back when I first started I'm pretty sure I would be wondering if all this work would actually pay off. I guess it is already the case that people wonder if the effort of writing an application is going to pay off, and this is even more effort. When you first start publishing it is not clear what all the benefits might be, so the real road test equipment may be the only motive for a new applicant. If it is considered to be a lottery with low probability of "winning" it won't appeal to everyone.

                            2 of 2 people found this helpful
                          • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                            neuromodulator

                            I think maybe roadtesters could be divided into tiers, lets say 4 levels:

                             

                            tier 0: Is the level of any new member.

                            tier 1: Is reached after writing a good blog post (ideally a review), this would show that the member can actually write something for the community.

                            tier 2: is reached for roadtesting an item that costs <100 USD

                            tier 3: is reached for roadtesting an item that costs <1000 USD

                             

                            Something like this would probably increase the number of written reviews, not to 100% but probably closer.

                            1 of 1 people found this helpful
                            • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                              gecoz

                              Randall,

                               

                              I think the program is working quite well the way you have shaped it so far, and this is definitely credit to you It is frustrating that, despite all the efforts and the good will placed to try keeping the program as open to newcomers as possible, you get this kind of response. But, as pointed out in other discussions on the topic, I think those cases of "missed reviews" do not detract from the success of the program, as it is unrealistic to expect a 100% return for all the roadtests.

                               

                              For securing a sustained success for the roadtest program, it is important it is perceived as an open program, hence I believe it is vital that new blood is continuously brought on board. As I already had chance to explain, I fear that making the application process too hard or convoluted, or introducing a sort of "rite of passage" is a risk: lets not forget that many potential roadtesters hold a full time job, and need to be attracted to the program somehow. If they feel they need to invest too much time, they might just not bother trying at all.

                               

                              As I said, I feel the program is working quite well as it is, and your statistics seem to confirm that, so, ultimately, I think you should resist any urge to change the application process, especially if such change could involuntarily end up creating a preferential route for the experienced roadtesters. And if the only good applications you receive are coming from the experienced roadtesters, perhaps I would still try to contact a few newcomers, and ask if they can expand on their application. If they show interest and commitment, then you could decide if any of them is worth the risk.

                               

                              Here obviously I'm not talking about the people that are out to game the system: no matter how easy or hard you make the application process, there will always be people that will play it to their advantage, so trying to shape the application process to weed-out such people ends up being  counterproductive for all the rest of genuine applicants. All you can do is make the suppliers aware of the risk (they probably already know that anyway).

                               

                              Thank you for all the hardwork you put to make the roadtest program such a great one!

                               

                              Fabio

                              4 of 4 people found this helpful
                              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                                jomoenginer

                                That is a shame that you took a shot to appease someone but then get burned with no RoadTest reviews.  I'm curious, was the person that had complained about only experienced RoadTesters getting selected one of the 4 that you took a chance with? 

                                 

                                I'm sure there are good folks that are either new to the site or had seen the RoadTest posted on something like LinkedIn or another site who are more than willing to produce a good review, yet they may not get a shot now due to the actions of the 4 you mention.  Outside of doing it like the hotels do by requiring a RoadTester to supply a credit card number before receiving the product, there must be another means to screen a potential RoadTester.   Perhaps requiring them to have participated in something like a Project14 contest or have posted a certain amount of content on the site, or a reference to postings elsewhere, before being eligible? 

                                6 of 6 people found this helpful
                                • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                                  aabhas

                                  rscasny

                                  I agree to jomoenginer that you should have data of any payment method so that if the roadtester is not able to complete the review without any genuine reason then he has to pay for the hardware , this would eliminate participation of all the members who want to take part just for hardware and not fulfill responsibility of writing the review .

                                  But you should not remove participation of the new members rather see their participation in the community and get to know about them and you can even take help from tariq.ahmad  and sort out that who are taking part in project 14 , or any other contests and their credibility to be given the hardware or not .

                                  I am saying this because I am myself a new member to the community and have participated in Project14 but have not at all participated in RoadTests & Reviews  but will also take part here and I think this is also a great exposure and platform for roadtesting writing the official review of a commercial product and I think that you definitely is a risk giving new members a chance then you should have  less ratio of new members bieng declared roadtesters  but you should not remove most participation of new members as many members like me may be seeking to try their hands on this part of the community along with dedication.

                                  Though I am no one to tell you as I don't have the responsibilities what you have and you might have some target that you have given to the manufacturer/sponsor about the reviews bieng received but this is my personal  opinion.

                                  Thanks

                                  Aabhas Senapati

                                  1 of 1 people found this helpful
                                    • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                                      shabaz

                                      Hi Aabhas,

                                       

                                      You've made effort with Project14, so you're not a new member any more : ) Your thing finder project was interesting : ) And you made an effort to understand how to use the tools, like the code syntax and insert image features in the online editor.. so many people don't. Even really skilled engineers can have difficulty communicating information (partially stereotype, but partially true) , so Project14, the RoadTests, and other activity writing could be great ways to get an edge when job-hunting.

                                      2 of 2 people found this helpful
                                        • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                                          aabhas

                                          shabaz

                                          Though I have participated in two competitions and have been selected for one more competition Project14 but have taken part in the element14 community from January 2019 which is very less in comparison to senior members and the experience you all senior members have by being in such a wonderful community for many years . No one can become an overnight star and it is a gradual process so I have to work harder and gain a lot more knowledge but I really admire a lot about this amazing community it has provided with a lot of exposure to me and learn a lot of new things and the community members are also very helpful and are ready to help even when I ask them personally. Well, I am still a student and have to manage my studies and exams as well but I think this experience and knowledge I am getting will definitely help me in future so I try to invest my time here from my present time and try to participate in Element 14 Community. This time will try out my hands on the RoadTests & Reviews community and try to submit applications.

                                          Thanks

                                          Aabhas Senapati

                                          2 of 2 people found this helpful
                                      • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                                        Jan Cumps

                                        This is not too different from dealing with anyone new in other circumstances.

                                        At my job, I hire contractors without meeting them in person too - and there's no 100% guarantee of satisfaction (also not when you interview staff on location).

                                         

                                        I admire Randall's effort to get new reviewers. Kudos for not taking the easy path. It's the best way to keep things healthy and to grow the community. Saying that, it's not easy. There's no path to success.

                                        The only thing I can advise is: take the risk. Allow for failure. Assign low risk items to as many new reviewers as you can.

                                         

                                        Settled reviewers aren't saints either. I've seen lousy reviews and no-shows from settled members.

                                        4 of 4 people found this helpful
                                        • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                                          Fred27

                                          shabaz  wrote:

                                          Even really skilled engineers can have difficulty communicating information (partially stereotype, but partially true) , so Project14, the RoadTests, and other activity writing could be great ways to get an edge when job-hunting.

                                          I agree that things like road tests and Project 14 are really helpful with expanding skills that you can use in your job (or to get a new job). Some stereotypical engineers might need these skills more than others, but we can all benefit to some degree. It shows that you can research and understand a topic or piece of equipment, and then pass on that knowledge to help others.

                                           

                                          I'm a software developer, so electronics and embedded development isn't directly related to my job. However, I've done presentations to my colleagues on these sort of subjects - often at lunchtime with pizza provided. (I even ended up repeating one of these presentations on stage at Microsoft's UK HQ at a hackathon. Nerve-wracking stuff!) I've done others that are a little closer to work topics too. It definitely helps you when you want to get ideas across to management or prove you're the guy to promote. They fit well on your CV as the extra stuff that proves you have a bit more to you that just the skills they listed in the job spec.

                                           

                                          I'll give you an example. Right now I'm wondering whether my C# web developer colleagues who are familiar with Azure would be interested in finding out more about Azure Sphere. They are unlikely to have even heard of it. If I road test one, then that might form the basic of a presentation. Should I decide to change jobs, I'm very likely to be discussing Azure in a job interview. Having published an article on an aspect that the other candidates have never heard of - that's bound to go down well.

                                           

                                          Having links to things like road tests that you have done outside work is definitely a good thing. Even if you don't intend to show these to anyone outside the E14 community then you've built a valuable skill that will help you a lot. To any new member who's reading this then I'd say that it's really worth going for a road test and it's really worth putting some effort in - before you start as well as after you're selected.

                                           

                                          Anyway, getting back on topic...

                                          4 of 4 people found this helpful
                                          • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                            ciorga

                                            Hi Randall,

                                            I like what you wrote in this message.   This issue was what made me stop applying to roadtests (specifically the issue that happened with the R&S FPC1500 spectrum analyzer roadtest).  I may start applying again.

                                            Best Wishes,

                                            Cosmin

                                            1 of 1 people found this helpful
                                            • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                              robogary

                                              Thanks for the message, it reinforces that integrity does really count still.

                                              There will always be extenuating circumstances for some Road Testers, ......and always some pinocchios,......... hang in there, keep the faith for the 99%'ers.

                                              1 of 1 people found this helpful
                                              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews
                                                balearicdynamics

                                                Hi Randall,

                                                 

                                                I am so sorry to read this. The thing that leaves me really surprised is that there are many people like this than expected. I am the first that I missed some road testò this has been due to some my personal problems, but I know this is an excuse. And I had the opportunity to make other road test after this debacle and I appreciated this very much.

                                                I suppose what you need is a solution or at least a proposal. The idea may be that there should be a road test preview, before the deadline. I mean, after sending the first email recall in the mid of the road test period, if you have no answer or there is not draft already written by the roadtester, you can at least suspend him and do something before the expected period ends. This does not solve the problem that one piece of hardware has been tossed out but at least there is something that can be done respect the customer/producer involved in the roadtesting.

                                                 

                                                About the value and importance of road tests, I have the intention to mention the roadtesting opportunity as a great free resource for makers in one of the chapters of the book I am writing, I have already sent you a message about this. take a look and let me know if it is worth to have a contribution by you.

                                                 

                                                Enrico

                                                3 of 3 people found this helpful
                                                • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                  rscasny

                                                  Hi All.

                                                   

                                                  I'd like to thank everyone who has participated in this discussion. It's help me clarify the problem and think about solutions.

                                                   

                                                  Change Idea #1: I used to send a standard availability email when I notify the roadtest finalists wether they are new roadtesters or experience. I will start sending out a separate email for new roadtesters that spells out the responsibilities of a roadtester and the consequences of not doing a review.

                                                   

                                                  Change Idea #2-- I am thinking that I should have an onboarding webinar for new roadtesters. It may be a way to build relationships with new people and they can hear from me and what I expect.

                                                   

                                                  Change Idea #3--I have had some instances when a member was not available because he was doing another roadtest. I am considering setting up later due dates for these people instead of choosing someone else who wrote a below average application.

                                                   

                                                  Change Idea #4--If I got 5 kits to hand out and I honestly feel I don't have 5 reliable roadtesters, I may have to select less than 5 and reserve the other kits for a project.

                                                  8 of 8 people found this helpful
                                                    • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                      Fred27

                                                      They all sound like good ideas. I don't think there would be any harm suggesting some criteria that might help get you selected though. "Preference given to members who have watched and commented on the onboarding webinar", "Preference given to members who have already written a blog about a development board". Something like that. They can all be at your discretion, but a hint at some expected prior effort.

                                                       

                                                      I've also been wondering if there's any way experienced road testers could mentor those who want to be selected. However, if the problem is a lack of effort rather than a lack of skills on the part of the missing road tests then I don't suppose there's much that can be done.

                                                       

                                                       

                                                      I notice that the Azure Sphere dates have been pushed back a bit. I hope that doesn't mean that of the 359 applicants there weren't 50 good ones.

                                                      2 of 2 people found this helpful
                                                      • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                        balearicdynamics

                                                        All the change ideas are good IMHO. But

                                                         

                                                        Change Idea #2-- I am thinking that I should have an onboarding webinar for new roadtesters. It may be a way to build relationships with new people and they can hear from me and what I expect.

                                                         

                                                        This is the best I think. And if you want to involve some of the already known road testers / top members etc. maybe this can be a good moment for create a community relationship. Anyway this sound very good.

                                                         

                                                        Enrico

                                                        3 of 3 people found this helpful
                                                        • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                          colporteur

                                                          CI#1: This has merit. The new parties know the rules going in. The more they know, the easier it is for you to take action. What consequences can you enforce?

                                                           

                                                          CI#2: This idea has some merit. I am assuming the completion of the webinar for new applicants can be tracked. I suggest once they are accepted, you require them to complete the process a week before shipping the goods. Failure to comply, you can withdraw their names.

                                                           

                                                          CI#3 is valid. It is not unreasonable if a person is committed to a RoadTest, the scheduling for another RoadTest can be delayed. You can make that decision depending on the commitment to your vendors.

                                                           

                                                          CI#4 I'm going to assume no RoadTest without product distribution is better than no RoadTest and product gone with no return. If you send kits to marginal participants, then you are putting yourself at risk. How you can use the product for a future test, is up to you. I rather liked the grab bag RoadTest, I got selected for.

                                                           

                                                          How long do you have to be a member to be eligible? I think a cooling down period may help screening, especially if advertisement outside this site is attracting undesirables. Active members for <30 Days need not apply.

                                                           

                                                          Is there someway to create a minor league, that players start out in and work their way to the big leagues. Or maybe a tiered RoadTest participation roster. The vendors can help select the tier they want their product tested at. Their is a junior, minors and professional levels. Maybe you have to work the AAA "triple A" club (i.e. webinar, why me plea and other obstacles) before getting to the juniors. From there you work you way up to the pro's.

                                                           

                                                          I'm thinking the mission badges must be able to help you there. You have people training in the minors for a shot.

                                                           

                                                          Just spit balling ideas while I am waiting for the Canadian world cup of soccer womans game to start. Go Canada!

                                                           

                                                          Sean

                                                          • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                            lui_gough

                                                            Dear rscasny,

                                                             

                                                            Some interesting ideas for solutions:

                                                            Change Idea #1: A good idea, low implementation cost, but perhaps won't do much as initially, everyone will be "willing". Even with the standard e-mail, your new RoadTesters would have at least replied to accept the item ... before going silent later. I suspect it may not have much of an effect in the end.

                                                             

                                                            Change Idea #2-- Perhaps a more difficult idea to implement, as many of us RoadTesters are scattered all over the world, thus time zones can be an issue along with internet connectivity. It is, unfortunately, why I've ended up not attending many of the sponsor's webinars even though I would have liked to. I say that from the perspective of someone that's on rather unreliable quota limited LTE (~20GB/month) who often has to curtail video entertainment to make it through the month. Holding a video conference can be a little difficult for some people.

                                                             

                                                            Change Idea #3--This might be a good idea, but I'm concerned that the sponsor may not always be willing to have "late" content appear as the marketing value might not be there. Furthermore, some of the RoadTesters who are also reviewing the product may misunderstand the special deadlines awarded and incorrectly criticise an established RoadTester. There is also the loss of resources in the sense that when all RoadTests for a given product run in parallel, we can consult other RoadTesters for suggestions to overcome problems. This may not work as well in the case of disjoint start-end dates. Another potential problem is that later RoadTesters may have their impartiality compromised if they view or are influenced by already published reviews.

                                                             

                                                            Change Idea #4--I think you've probably been doing this to some degree already. I don't see this as a bad thing - there's no point awarding to applicants who don't meet a minimum standard.

                                                             

                                                            I'm not sure there is any "magic bullet" solution to the issues, but these are just my thoughts.

                                                             

                                                            - Gough

                                                            1 of 1 people found this helpful
                                                            • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                              rusgray

                                                              As part of the onboarding webinar and/or email for new testers, it might be helpful to show/link examples of satisfactorily completed road tests to illustrate what you're looking for in a completed RoadTest review.

                                                              1 of 1 people found this helpful
                                                              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                gecoz

                                                                I think ideas #1 and #2 are good: any guidance and information about the program, how to apply, what is expected and how to prepare and deliver a RoadTest review will be of invaluable help for all the members, not just the new ones.

                                                                 

                                                                Ideas #3 I don't think it is a good one. I can see where you are coming from with this, and what you are trying to address here, but I think the RoadTest program should have few and very clear rules, and one of them should be the start date and the end date of the RoadTest itself. Starting moving dates to accomodate members schedules can cause confusion and lead to odd situations.

                                                                 

                                                                Idea #4: to some extent I believe this is already happening, so I don't see it as a bad thing in itself . The problem is that you might end up not taking any risk at all on new members this way, and that could be a problem. As already pointed out by many fellow members and yourself, writing a good review, unless you are particularly gifted person, is a skill that requires lots of "trial and error" attempts, and lots of feedback. Writing a not -so-good application doesn't tell much on the real potential of a candidate, just like writing a very good one is not a guarantee of success. Even with interviews, sometimes you cannot find out what to expect from a candidate, so ultimately the only way is to take a risk, trust your instinct and choose some candidates regardless. Not choosing, to play safe, is a bad bet.

                                                                 

                                                                So, in short, I'm in favour to any change that introduces clarity in the process, helps members understand what is expected from them and encourage participation. The other changes proposed, I'm not sure what they aim at. If it is to reduce the risk of "no-show" review, I'm afraid they won't make much difference. If it is to try getting good quality reviews, then I think  perhaps, for clarity and transparency sake, could be more useful to resurrect something like the RoadTest Plus program, where you can invite experienced members to do a review. For some (or all) RoadTests, you could run a RoadTest Plus (project based rather than just test based) where all the unassigned kits from the regular RoadTests are made available.

                                                                 

                                                                Personally, as I said, I think the program is working quite well as it is now. I don't know what bargaining power you have with the suppliers, but I suspect it could be possible to make sure that they agree with assigning, for example, 75% of the kits for regular members, 25% expressly reserved for new members. That way, you can manage all the stakeholders expectations while making the program rules as clear and transparent as possible and chances are given to new members.

                                                                 

                                                                One last change I would love to see: some feedback from you or the supplier about the published review. I understand it would generate more work for you and the supplier, and it could be a bit of double-edged sword, but for all the members that genuinely are trying to do a good job of their review, this feedback would be invaluable. Preferably, this should be delivered as private conversation, to avoid any problem. I know, for one, I would love to receive such feedback, to understand where I need to work more, in order to improve.

                                                                 

                                                                That's my 2p.

                                                                 

                                                                Fabio

                                                                1 of 1 people found this helpful
                                                                • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                  gpolder

                                                                  Hi rscasny,

                                                                   

                                                                  I like all the change ideas and would like to propose an additional one:

                                                                   

                                                                  Selected roadtesters should write an introduction and testplan that goes further than the application, prior to sending out the equipment.

                                                                  3 of 3 people found this helpful
                                                                    • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                      Fred27

                                                                      That's not a bad idea. One thing I like about the new application template is that you can get your proposal across without having to write loads. It saves everyone time - especially Randall reading all of them. However, I always do some research. I'll often have an IDE installed before the items turn up, etc. I'd that's part of my road test I'll often start writing or making notes. Formalising this might be helpful and if the road tester had done done of the work already they might be keener to continue.

                                                                    • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                      dougw

                                                                      I think there may be other factors that might be useful in choosing applicants.

                                                                      • one has to do with matching road tests to people who can handle them. Specifically predicting how hard a particular road test will be to complete, given a particular skill set or sometimes any skill set. I often try to predict what the response rate will be for each road test and find the predictions are significantly better than just random chance, so predictions might be useful in deciding how to stack the deck.
                                                                      • part of the application could be a requirement to estimate how many hours will be spent on the road test and blog. (separate numbers) I am sure it would be useful to rscasny  if an applicant spent some effort to explain how much time they are willing and able to commit to a road test.

                                                                      I think Randall is doing a superb job screening, filtering and encouraging applicants, but there are some road tests where it will always be problematic to get a high completion rate.

                                                                      • Part of this is due to applicants not honoring their commitments but this is not the only reason
                                                                      • When I look at a new road test, I sometimes think things like: "wow - there aren't too many members who will be able to handle this road test" or "yikes - this road test will require a scary amount of work" or "this is new and different with rudimentary support materials so it will take a lot of learning and perseverance". When I get those types of reactions, it almost always means there will be few completed road tests, partly because other experienced road testers will shy away from applying due to similar reactions. The road test may scare off knowledgeable road testers leaving only naïve applicants that rapidly discover they can't handle it.
                                                                      • Part is due to the nature of the specific equipment being road tested. The manufacturer may not have made it easy enough to do a road test.
                                                                      • Sometimes the equipment is so esoteric, the average member doesn't have a use for it and is not very motivated once they realize this.
                                                                      • Sometimes the road test is just very hard and even with the best intentions, success is not likely (for some, blogging about a failure is super hard)

                                                                      The bottom line is when there is a poor response rate, don't automatically assume it was due to poor applicant selection criteria or poor screening, there are a lot of factors that will occasionally conspire to yield a poor result. Sometimes it is even predictable before the candidates are selected.

                                                                      It is worth examining each anomaly, but some of it is just statistical reality.

                                                                      2 of 2 people found this helpful
                                                                        • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                          luislabmo

                                                                          Hello Doug,

                                                                           

                                                                          I agree with your points about matching a RT to an applicant with the right skill set required and also agree that Randall and his staff do a remarkable job with the RT program but, when a reviewer doesn't proactively seek for help -in case of issues-, hasn't produced any kind of content within the required 60-day period or even worst, doesn't respond to follow up requests I believe in the majority of the cases these all point to the same direction.

                                                                           

                                                                          Luis

                                                                          2 of 2 people found this helpful
                                                                      • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                        shabaz

                                                                        Incidentally, if the particular RoadTests  are for equipment we may likely have, or are low-cost (I still have $200 Project14 prize budget to use up) I'd be happy to do a quick review if it increases the chance the manufacturer will be happy for future RoadTests too, otherwise it is unlikely they will want to do it again I guess : (

                                                                        6 of 6 people found this helpful
                                                                        • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                          Gordon Margulieux

                                                                          Randall,

                                                                           

                                                                          I’m sorry for jumping in late on this discussion, but I’ve been very busy lately and of course all the E14 network issues. 

                                                                           

                                                                          First, I would like to thank you for all your great work.  I don’t envy your job.  I have worked with high school students for a while and what they try to pass off for homework is sad sometimes.  What is more amazing is how mad they get when you don’t accept their work. 

                                                                           

                                                                          Second, I may have missed it, but you didn’t mention what sorts of roadtests were not being completed on time.  There can be a lot of unexpected work involved in any roadtest. I have noticed some reviews appear relatively late.  As some have already mentioned, on high ticket items it would seem fair to add to the rules for roadtests that until you successfully completed a number of smaller roadtests you would not be legible for larger items.  Do you ever have to contact a reviewer to tell them their review was not satisfactory?

                                                                           

                                                                          Third, I know you mentioned that you had tried to contact those you who have not completed reviews, with some not replying.  I have also noticed a few reviewers that don’t enter their reviews using the form on the product page, but through the roadtest blog page.  Their reviews never seem to show up as completed. 

                                                                           

                                                                          Thanks again for doing a great job.

                                                                           

                                                                          Regards, Gordon

                                                                          1 of 1 people found this helpful
                                                                          • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                            luislabmo

                                                                            It is sad to hear that after the great efforts from the element14 staff things like this still happen to a great program.

                                                                             

                                                                            I mainly came to say that I agree with Fred27 and other members that advertising free products as a way to attract people to the community -which understandably is an effective marketing formula, is probably not a good approach IMO. Where I come from there is a saying "the first impression is the one that counts"; if a person joins the community (or participates for the first time) because of the "you can get this for free" ad it will inevitably attract freeloaders adding more risk of failure. This not only impacts the RT program, this also impacts the programs like Design Challenge where you can get free stuff in advance and then vanish if that was your intention. This is evident in some profiles where you can see the members just created the account to apply for the free stuff and stay silent without any contribution to the community.

                                                                             

                                                                            I do agree that strategies that keep the Road Test program open to newcomers is something we all benefit from -I myself was a new Road Tester once, new comers, marketing, readers and e14 will benefit from this as well, I consider useful and perhaps regular contributions to the community should be a must before starting in the program and I also agree with others that in order to test the waters members should start with something cheap and/or easy as a way to prove that a quality review can be delivered on time before handing more expensive/complex/specialized equipment.

                                                                             

                                                                            Luis

                                                                            1 of 1 people found this helpful
                                                                            • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                              danb1@shaw.ca

                                                                              I know I haven't really posted much here on this community in the past but I read this post and just wanted to provide some insight as to what I have seen in the past for companies that I have done beta testing for in the past and I feel that these methods may work here if you are looking for a method to thin out those that aren't serious about providing acurate and or honest reviews. Please see the poorly formatted list of methods if seen before:

                                                                               

                                                                              1) Company XYZ provides the ability to test a product with high value that must be returned at the end of the test for "modifications" to a new user they are told that they must provide a deposit that will be returned after the success of the beta... experienced and proven testers don't have to meet this requirement... (as a new tester I wasnt happy about the deposit method but for items that really peaked my interest I dealt with it and put the deposit on my credit card or to my PayPal...and did the test and as soon as the company received the confirmation from FedEx that the package was received the funde were promptly refunded. This I'm sure kept the dishonest from applying)

                                                                               

                                                                              2) (my favorite method) The testing company requires everyone to complete a review/test on a website that has known issues... I'm not sure how many there are but they look for people to find and report on that and then if they have done so to a good standard they are selected for future tasks... if not some feed back it provided and they must wait a month before reapplying. This helps weed out the people that tend to not provide quality results.

                                                                              2 of 2 people found this helpful
                                                                              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                fmilburn

                                                                                Hi Randall,

                                                                                 

                                                                                I will put forth an experience from the opposite viewpoint.

                                                                                 

                                                                                My own experience is that I was not chosen as a RoadTester on early attempts although I feel they were good submissions and topics I had familiarity with.  While not selected on my first attempt I went ahead and did my project anyway with a board I bought.  In other words I really was serious about doing it. Of the 3 persons actually selected (all of them level 1 as was I), none actually wrote up a RoadTest.  This was before you were managing the program but it actually dissuaded me from making entries for a while....

                                                                                 

                                                                                I was not selected until after I had become active and in retrospect this seems reasonable. In general I believe extra weight should be given to those who are active on e14 and a statement to that effect could be included in the instructions. If the company making the product has specific requirements for testers (like current job or educational /work background) that should be stated also.  This might help RoadTesters with their expectations of being selected and give incentive to become more active.

                                                                                2 of 2 people found this helpful
                                                                                • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                  dougw

                                                                                  Suppose Randall has increased the probability of completed road tests to 80%. (Well done Randall)

                                                                                  This still means 1 out of 5 will not complete their road test.

                                                                                  So if there are 4 road testers, what is the probability that none will complete their road test?

                                                                                  This is simple probability theory, but my probability theory is super rusty, however by my calculation this means

                                                                                  1/5 x 1/5 x 1/5 x 1/5 = 1/625.

                                                                                  So even if you have an 80% success rate, every 600 road tests or so you could have a road test with no completions, and it could happen at any time - the 6th road test or the 600th road test.

                                                                                  This is just an example and of course many other factors can affect the rate, but the statistics alone say sometimes the whole road test will flop .... It is unfortunate but it doesn't automatically mean the wheels fell off or something went terribly wrong.

                                                                                    • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                      lui_gough

                                                                                      Some very dangerous assumptions dougw ... you can't take an aggregate rate (i.e. 80% completions on average) and then apply it to individual RoadTesters as it's not a game of equal likelihood, especially when those who fail to deliver are disqualified and those who have delivered in the past may have greater motivation to maintain their record. If all events are independent with no biases (i.e. assuming the mix of candidates stays constant and Randall picks randomly without any "tiering") then perhaps your calculation could hold, in a rough back of the envelope kind of way. Looking at our own track records, we can see that we're holding up the high-end of the bar. Somehow I think that having zero returns on a RoadTest is highly improbable, although perhaps for kits with just one or two candidates, this may be more likely. I have some faith that some people would deliver a report just to be "the first" to do so ... they often get more views!

                                                                                       

                                                                                      But more likely than not, external influences may well be the dominant part of the equation:

                                                                                      • New members without much experience vastly underestimate the effort, equipment needed to deliver on their proposed test plans (perhaps due to a lack of research). They may run into an insurmountable hurdle and perhaps get discouraged - maybe even during the write-up. Maybe they see someone else has already done everything they said they would, and did it even better, thus they don't deliver.
                                                                                      • The product itself isn't interesting enough on its own to warrant further time investment upon meeting one of these hurdles. Or perhaps they decided to register anyway to have a go, being too optimistic and ignoring the consequences and then get cold feet or lose confidence.
                                                                                      • They never receive the product, or it gets damaged in transit and they can't be bothered to chase up.
                                                                                      • The product goes out much later than initially assumed and then other commitments get in the way - this has happened to me a few times but I've worked hard to make sure I get around to it. Other people may not be as lucky.
                                                                                      • The product goes out at a time when people may be busy with other things (e.g. holiday periods).
                                                                                      • The person chosen may not have the same sense of responsibility or may have other complicating issues (i.e. health of themselves, close family members, etc).

                                                                                       

                                                                                      I think that perhaps focusing on completion rate is missing the other part of the equation - review quality. It was mentioned prior that feedback from the supplier could be solicited - but I think it may even be useful just to get feedback from element14 themselves just to know what they think about our performance. Delivering no review and delivering a substandard (or blatantly incorrect) review are perhaps not that far apart on the value scale.

                                                                                       

                                                                                      - Gough

                                                                                      2 of 2 people found this helpful
                                                                                        • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                          dougw

                                                                                          I wasn't trying to be dangerous It is just a simple hypothetical example illustrating that if you don't have 100% completion rates, sooner or later a lot of non-completions can happen at the same time.

                                                                                          We have been speculating extensively about why these results occurred, without much real root cause information, and we theorize what can be done to eliminate these types of results. It is all interesting and possibly useful but it isn't definitive. I don't see the harm in also speculating that statistics might provide some insight and a different perspective of what happened. If it looks useful, then collecting some real data might provide some interesting statistical analysis.

                                                                                            • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                              lui_gough

                                                                                              I wasn't being serious ... but hey, the mis-application of statistics in everyday life seems to be a pet peeve of statisticians so I thought I'd throw it out there. In all, we do have several things going on - things can often break down when we examine very small numbers (e.g. a RoadTest with just 1 or 2 kits on offer), so applying aggregate figures would be unwise. Likewise, where there are non-equal chances of selection, that often will throw things out the window, not to mention that the pool of applicants will depend on how "interesting" a device is and the sponsor's requirements.

                                                                                               

                                                                                              In the end, statistics is still why some people can still rock-up to a casino and play roulette, putting everything on red with a ~47% chance, play once, win (and never come back) even though statistically, the odds were pointing towards a loss .

                                                                                               

                                                                                              - Gough

                                                                                              3 of 3 people found this helpful
                                                                                                • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                                  colporteur

                                                                                                  Please take this commentary as jocularity for which was written. To those still employed by the machine, not so much.

                                                                                                   

                                                                                                  My employer had come up with a strategy to save each member of the team 5minutes. The bean counters set about adding up the time saved and determined we didn't need to hire an additional person that year through the savings. I will venture to guess anyone working in corporate, has witnessed the application of this theory and no doubt has given their head a shake.

                                                                                                   

                                                                                                  At the next group meeting I tabled a facetious suggestion. I asked all members of the team, including myself, to cutoff one finger and donate it to the bean counters. I felt they were so go at making something out of nothing, they could make a hand with the severed appendages and attach it to their chest. Who hasn't been in the situation where you could use an extra hand.

                                                                                                   

                                                                                                  Sean

                                                                                                  1 of 1 people found this helpful
                                                                                          • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                            meera_hussien

                                                                                            Dear Randall and Roadtest Community,

                                                                                             

                                                                                                                       My reply maybe late. Sorry for that.

                                                                                             

                                                                                                                          Talking about this roadtest, i think it is a good initiative by the element 14 and the sponsor. Personally i think that roadtest has given chance to the member of the community to put their experience to test a new tools/equipment/development board, and share it to others. Without this roadtest, I dont think this oppurtuniy were available before. I am not sure how many will agree with me but this is very true for me. But it is sad when people are selected as the roadtesters and they failed to do the review. Because there are many other members who didnt get the chance. Hope every roadtester will apply for the roadtest with the intention to share the review with others and not for just getting a free product for nothing in return.

                                                                                            1 of 1 people found this helpful
                                                                                              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                                colporteur

                                                                                                I rather enjoyed your commentary. It gave me pause to think.

                                                                                                 

                                                                                                What return does a RoadTest deliver to the participant?

                                                                                                 

                                                                                                A quantitative response would be free stuff. That's easy to measure. Given some of my own thoughts, your answer implies something deeper. A chance to develop or apply skills. Not all technology careers offer the same opportunity. RoadTest can be that chance to explore. To do something you love to do yet can't for what ever reason. Maybe that is why we are so disappointed when our applications are not successful?

                                                                                                 

                                                                                                I believe some of the failures in RoadTest deliveries can be an effect from the very same cause. You are attracted to the shiny RoadTest widget. Emotional excitement takes over. You generate an awesome RoadTest application to impress yourself and of course the judges to secure the shiny bobble. After you are awarded the project, you discover that your enthusiasm has generated a RoadTest that is beyond your capabilities. Now you must face the truth or make a choice to ignore it.

                                                                                                 

                                                                                                Hey, I got the shiny bobble from the fools they were, what do I care. For me that is frustrating and disappointing. I wish you all the best on the path you have chosen. You missed a chance for your own greatness. There was something in the exercise you failed to learn, you didn't even realize that?

                                                                                                 

                                                                                                As I accumulate more and more of my trunk rings in age, I find myself examining my motivation for doing something. Many times I try and quantify my answer so people can understand. It seems, the accumulation of free stuff everyone understand. I doesn't rate high on my list. Self actualization and how my actions reflect back to me, who I am, seem to be more important.

                                                                                                 

                                                                                                If your desire is to be a published writer, one of the first things you are going to have to master is the proposal. Next is rejection of the proposal. Now wash, rinse and repeat that cycle maybe a thousand times. Do you still want to be that writer? The writer prerequisite (i.e. sales pitch/proposal) is not unique to writing, notice that same skill is transferable to becoming a RoadTester. Proposal/rejection appears to be a common theme in human lives.

                                                                                                 

                                                                                                I got to stop this I am rambling. Thanks for sharing your story. You may be able to tell from this commentary, you were never late to the party.

                                                                                                Sean

                                                                                                2 of 2 people found this helpful
                                                                                              • Re: 2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review
                                                                                                starbasessd

                                                                                                I have been an IT Help Desk for MANY years, Level 1, 2 and 3. I prefer the description 'jack of all trades, master of none', as I do electronics, woodwork, plumbing, etc. As such, I've been the lead on many projects for hardware, software and IT processes. Anywhere I've been project lead for changes, I've located several regular and some new people for each to do evaluations and testing.

                                                                                                Not everyone is good at it, but all have provided valuable information, even if it's 'I shouldn't have picked that person'.

                                                                                                I personally look first for someone who has made comments (good, bad, indifferent) about the current used hardware, software, or IT process. These people usually have skin invested in anything new or different coming down the line.

                                                                                                In this case, I might pick 75% for previous reviews or comments or known example projects, and 25% new people. I would ask of the new people what use case would they expect to try the new hardware for? That way you immediately have an idea as to what they have in mind, rather than leaving it totally open because they see a 'new, shiny' to get.

                                                                                                I might consider offering a template to follow, asking, for example of an RPi4 how well it works for WiFi, attached devices, overheating, throughput, etc. If the use case was as a tablet, how well it recognized the display, quickness of response, etc.

                                                                                                Then you could request a status every x number of days, questions for / from the manufacturer, etc, and hopefully a review after the time period. I'd also have available mentors or SMEs (Subject Matter Experts) to assist when needed, sometimes people might have no clue where to go when something isn't quite right...

                                                                                                Myself, I'm a hardware geek. I have all the PIs back to the beginning. I've used them for things like Asterisk, to Ham Radio, to motionEye, to weather reporting twitterbots.I'm looking forward to the Pi4 being available in the US so I can try it with all of the above, especially with comms requiring the true gigabit ethernet and USB3. I am starting to gather hardware for LoRaWAN to go with my Ham Radio toys. I have a couple of Elmer's (people who are SMEs in different areas) to push me in the right direction, and I try to help when and where I can as payback.

                                                                                                I feel your pain, and I've BTDT GTTS. I hope you are able to work it out.

                                                                                                2 of 2 people found this helpful