91 Replies Latest reply on Nov 12, 2017 7:30 PM by fyaocn

    Please Read: An Important RoadTest Program Change

    rscasny

      This is the third virtual "fireside chat" that I'm writing regarding the RoadTest Program. For this one, I'd like to roll out a RoadTest Program change and explain why the change is occurring.

       

      To start, I'd like to share a snippet of an email I received yesterday from a vendor who participates in the RoadTest Program:

       

      "We received only 13% reviews back from the RoadTests we sponsored in 2016. I would like to know the status of each of these roadtests. Are the missing reviews still being pursued? I would also like to know if the members that failed to post a review are now being flagged so we do not chose them again."

       

      It was not pleasant to respond to. I had to eat "humble pie," as we say in the US, when I responded to this vendor.

       

      To solve the problem, I decided I don't want to go down the road of suspending members from the RoadTest program. Suspensions sound too much like the hard rules of the military. I was in the Navy for 10 years and I didn't like tough rules back then, but I knew they were necessary given the reality of uncomfortable shipboard conditions and potentially dangerous operations we were participating in.

       

      But this is a community. It should be a place to have some fun, meet new people, network professionally, be a mentor, or get mentored. Suspending people just is the wrong road to follow.

       

      The way I've chosen to resolve the problem is the program change.

       

      RoadTest application reports will soon include whether a specific applicant has completed all of his/her previous RoadTest reviews. I'm told by our developers this is very easy to do. We will provide this information to the vendor with a RoadTester's application. The supplier will make the decision where to select the RoadTester or not. Given the above comment by the supplier, I think the answer is self-evident.

       

      Now, this change will probably cause me more work. If someone were flagged, I will have to check to see if that person is on a current RoadTest and it's before the due date of the official review. I surely don't want to penalize a conscientious RoadTester. I will also have to ensure there are not any extenuating circumstances that have caused the person not to write the review, i.e., sickness, family issues.

       

      I am ambivalent about this whole thing. There are many RoadTesters who are doing a fantastic job. But a 13% success rate is not successful. I really have no choice but to implement this change.

       

      Sincerely,

       

      Randall Scasny

      RoadTest Program Manager

        • Re: Please Read: An Important RoadTest Program Change
          geralds

          Hi Randall,

           

          for me, I'm absolut agree with you.

          I think that many can be misunderstood  because many of Road Testers are not so fit with the English language.

          Me too a bit, because my language is German, so i translate difficult sentences every time with the (e.g. google) translator.

          I hope so much all of you can understand me with my jargon mishmash "Kauderwelsch".

           

          If i write here the review is this enough or have i write also via email to an email address?

          Because i never have seen some email address what you meant above -  "reviews back to the sponsors."

          How goes the review back to the sponsors? - with our review here written in element14 road test, or with an email from you?

          Are the sponsors reading here the reviews?

           

          Best Regards

          Gerald

          ---

          • Re: Please Read: An Important RoadTest Program Change
            rachaelp

            Randall Scasny wrote:

             

            To solve the problem, I decided I don't want to go down the road of suspending members from the RoadTest program. Suspensions sound too much like the hard rules of the military. I was in the Navy for 10 years and I didn't like tough rules back then, but I knew they were necessary given the reality of uncomfortable shipboard conditions and potentially dangerous operations we were participating in.

             

            But this is a community. It should be a place to have some fun, meet new people, network professionally, be a mentor, or get mentored. Suspending people just is the wrong road to follow.

            Hi Randall,

             

            I see where you are coming from but to my mind If a member who hasn't completed a particular RoadTest, hasn't contacted you to tell you in advance of their extenuating circumstances, isn't responsive when you contact them, etc, then I don't personally see the downside in suspending them from the program. There has to be some way to dissuade people from signing up just to get free stuff and if its the price we have to pay to ensure suppliers continue to offer their products for free in exchange for a review then so be it.

             

            I was going to say we don't want a few selfish people spoiling it for everybody else, but sadly it seems it's not a few, for this particular supplier it's 87% which quite frankly is shameful, and those people (unless they have a very good reason why they haven't completed their reviews) should be utterly ashamed of themselves.

             

            Best Regards,

             

            Rachael

            4 of 4 people found this helpful
              • Re: Please Read: An Important RoadTest Program Change
                kas.lewis

                I feel part of the problem of an outright suspension in that those selfish people (wether few in number or not) could generate a new account with ease. If you keep them here maybe they won't create a new account. It's a thought and I must admit not a very well thought out one, But there is always the chance of people creating multiple accounts for the larger more interesting road tests or after getting suspended creating new accounts and doing the same thing all over again.

                 

                K

                  • Re: Please Read: An Important RoadTest Program Change
                    rachaelp

                    Kas Lewis wrote:

                     

                    I feel part of the problem of an outright suspension in that those selfish people (wether few in number or not) could generate a new account with ease. If you keep them here maybe they won't create a new account. It's a thought and I must admit not a very well thought out one, But there is always the chance of people creating multiple accounts for the larger more interesting road tests or after getting suspended creating new accounts and doing the same thing all over again.

                     

                    K

                    I would hope that there was more to receiving expensive free kit than just a user account on this forum. I would expect there would be some sort of verification of who they are, real name, address, contact details, some form of ID as proof of who they are etc. Much harder to have lots of duplicates of all that information. I also don't think we were talking about suspending their accounts on here, just their eligibility for signing up for road tests.

                     

                    Best Regards,

                     

                    Rachael

                    2 of 2 people found this helpful
                    • Re: Please Read: An Important RoadTest Program Change
                      jadew

                      I suspected duplicate accounts too. People who enroll just to get their hands on the item, with no intention of giving back a review, would certainly have no moral issues with creating as many accounts as they can and enrolling with all of them.

                       

                      I've often wondered what these people could possibly say in their applications that they're sometimes chosen almost exclusively over actually qualified and willing roadtesters.

                      2 of 2 people found this helpful
                        • Re: Please Read: An Important RoadTest Program Change
                          jkutzsch

                          Interesting thoughts on multiple accounts.  Not knowing the logistics that Element 14 would have to go through, but perhaps a shipping address flagged list?  Of course this could be bypassed by multiple P.O. Box use, etc...  But depending on programming resources available I would think cross referencing accounts, shipping address and even IP might allow an interesting source to provide basic tracking.  Obviously having Randall monitor individually could turn quite brutal for his time, but if the bits could be captured and placed into a tool E14 can use, then perhaps that would be of assistance.

                          3 of 3 people found this helpful
                      • Re: Please Read: An Important RoadTest Program Change
                        rscasny

                        Rachael,

                         

                        You have a sound argument. Clearly, it appears some folks consider RoadTests a giveaway program, which it is not. We really need your professional opinions. My concern is perhaps if the delinquent roadtest reviews see they can make up their reviews, they can change their ways. We'll see how things play out. I need to improve the review percentage.

                         

                        Thanks for your participation in the community.

                         

                        Randall

                        4 of 4 people found this helpful
                          • Re: Please Read: An Important RoadTest Program Change
                            rachaelp

                            Hi Randall,

                             

                            Randall Scasny wrote:

                             

                            Rachael,

                             

                            You have a sound argument. Clearly, it appears some folks consider RoadTests a giveaway program, which it is not. We really need your professional opinions. My concern is perhaps if the delinquent roadtest reviews see they can make up their reviews, they can change their ways. We'll see how things play out. I need to improve the review percentage.

                             

                            Thanks for your participation in the community.

                             

                            Randall

                             

                            If this is the case they they will contact you or be responsive when you try to contact them. Any who are completely uncontactable shouldn't be cut any slack. You need to give them firm deadlines to get back to you when you contact them and let them know the consequences if you don't hear from them. For example:

                             

                            Dear XYZ, Your road test review of ABC is well overdue and I have not had any communication from you despite numerous attempts to contact you. I expect to receive either your submitted road test no later than DD-MM-YYYY, or an email detailing your extenuating circumstances no later than DD-MM-YYYY. Failure to do so will result in your suspension from all current and future road test programs and require the immediate return of any equipment you received. We will arrange a courier for the return of the equipment at our expense.

                             

                            If you don't do something like this I can't see how the situation can significantly improve and eventually suppliers will stop being willing to partake in programs like this.

                             

                            Best Regards,

                             

                            Rachael

                            1 of 1 people found this helpful
                          • Re: Please Read: An Important RoadTest Program Change
                            awneil

                            rachaelp  wrote:

                             

                            it seems it's not a few, for this particular supplier it's 87%

                             

                            Err ... without knowing the sample size, we don't know whether 87% is actually a large number of people.

                             

                            So it may be just a small number of people, and this supplier has been unlucky that they made up a large percentage of their testers ... ?

                             

                            Not that this excuses anything, of course - but we don't want to get things unnecessarily out of proportion.

                             

                             

                             

                             

                             

                             

                              • Re: Please Read: An Important RoadTest Program Change
                                rachaelp

                                Andy Neil wrote:

                                 

                                rachaelp wrote:

                                 

                                it seems it's not a few, for this particular supplier it's 87%

                                 

                                Err ... without knowing the sample size, we don't know whether 87% is actually a large number of people.

                                 

                                So it may be just a small number of people, and this supplier has been unlucky that they made up a large percentage of their testers ... ?

                                 

                                Not that this excuses anything, of course - but we don't want to get things unnecessarily out of proportion.

                                 

                                 

                                 

                                 

                                 

                                 

                                 

                                Even if this supplier is unlucky and their percentage failure rate is higher, I'm sure it's a problem across the board otherwise I suspect the focus would be why that supplier had such a low success rate and try to fix that rather than put changes in across the entire road test program to combat the issue so we can probably assume that, whilst it might not be as bad for all suppliers, it's a common problem.

                                 

                                To your point, yes we don't know the sample size but we know at least 7 people didn't submit a review for this one supplier. If I'd been giving product out for review and I only got results from 1 in 8 people I gave things to I would be quite annoyed.

                                 

                                Best Regards,


                                Rachael

                                1 of 1 people found this helpful
                                • Re: Please Read: An Important RoadTest Program Change
                                  rscasny

                                  It's a large sample I didn't want to spell it out because the actual numbers are dreadful. Simplifying it to a percentage is bad enough. But things are improving immensely. People are getting their reviews in or notifying about upcoming review postings, which was not happening 2 weeks ago. Thanks.

                                  4 of 4 people found this helpful
                                • Re: Please Read: An Important RoadTest Program Change
                                  balearicdynamics

                                  rscasny Randalll, when I was reading your original document I though just the same Rachael wrote. There are limits before we are obliged to manage situations IMHO. I agree that a military style enrollment is not the best for a community. But there are members that do senseless things and in this case it is the worth, probably at least to protect the other members that spend efforts and resources trying to be correct and helpful. Also this recent roadtest I participated by Infineon, I saw almost sadly that there was too few roadtest posted after the limit date.

                                  I appreciate that despite these happenings also in road test there is always space for new members or first-time roadtesters, So some action against those roadtesters that did not write down absolutely nothing (without a valid reason) is more than justified as they create a damage to all.

                                   

                                  BTW I think that one concept that probably is not well clear for many members is that the road tests are not challenges. But this maybe only my personal impressione.

                                   

                                  Enrico

                                    • Re: Please Read: An Important RoadTest Program Change
                                      rscasny

                                      Enrico,

                                       

                                      A couple of things have changed since I originally wrote this. (I think it's time for an full scale update; I'll put that on my work task list.)

                                       

                                      I have an assistant. I believe I introduced him to the RoadTest group awhile back. His name is Dan. He has been very helpful in doing follow up. With our launching more roadtests more frequently, I needed some help doing follow up. Dan's follow up emails have been not only good reminders, but we also discovered why some roadtesters had not posted. the two that come to mind was an illness in the family and someone lost his job and was trying to deal with it. (Once he started getting job leads, he quickly came up to doing the test.)

                                       

                                      The infineon roadtest was somewhat unique. I believe there were some board failures. for these people, I didn't feel it would be fair to them keep their deadline at the original due date. So, I extended it.

                                       

                                      People who did not do their roadtests for no good reason, I just can't use as Roadtesters. This delinquent group is getting smaller, however.

                                       

                                      As a group, the completion rate has increased quite a bit. So we are quite happy with that. Due to the quick turnaround, we are now feeling a lot more comfortable offering more roadtests because the roadtest group overall is unique and exceptional.

                                       

                                      More later...

                                       

                                      Randall

                                      3 of 3 people found this helpful
                                  • Re: Please Read: An Important RoadTest Program Change
                                    peteroakes

                                    I for one agree something needs to be done. I will not apply for a road test if I feel I can not complete it on the given schedule and also If I feel the award would be better served being given to someone else, and even then as was the case with the INA301 I may still post reviews or tutorials to help out the participants, its fun and most of the TMs like to help, it's what we're all about.

                                    I do get bothered though when an applicant gets a nice Scope or Analyzer or other more expensive product and then the next time you hear from them is when the next road test comes up., no review, no posts or anything in between

                                     

                                    I think your choice of dealing with this is very generous and hopefully will help weed out the collect and run types, it still won't stop the new members who sole purpose is to get a product and run, for this the only thing I have seen suggested in the past was to put min points limits on the "Better" product so if you have not been participating in the community then you dont get to review the "Big Toys" till you do.

                                     

                                    One thing that would also be bad though is if someone just posts a regurgitation of the product specs or simply a box opening, I have seen many instances in the past where the review is well to put it bluntly. Not a review. they go along the lines of "I opened the box...... it was a nice box...... The product looked like its picture and seems to meet the published specification....." you get the idea.

                                     

                                    So while I respect you measures, there may need to be more of a "Quality" check of the review, especially if from a very new member with no history of participation. A Post in the Review form does not necessarily mean a review has been done."

                                     

                                    That's my 2 cents (Pennies worth )

                                    4 of 4 people found this helpful
                                      • Re: Please Read: An Important RoadTest Program Change
                                        mcb1

                                        I have seen many instances in the past where the review is well to put it bluntly. Not a review

                                        I recall one where the comments were along the lines  .... "Is that all".

                                         

                                         

                                        Randall has a plan going forward so we should see where it goes.

                                        I'm pretty sure there will be one or two 'mistakes' but if thats all then it is a step up from the 13% rate for one supplier.

                                         

                                         

                                        Mark

                                      • Re: Please Read: An Important RoadTest Program Change
                                        awneil

                                        I think the enrolment form should highlight to the applicant that they have outstanding reports - and point out that this will count against them.

                                         

                                        It would also allow the applicant to explain any extenuating circumstances ...

                                        4 of 4 people found this helpful
                                        • Re: Please Read: An Important RoadTest Program Change
                                          gregoryfenton

                                          Having been chosen as a finalist for the first road test after this system has been in place (which I hope will have the desired outcome of more roadtests completed) I have an issue with the first question.

                                           

                                          Congratulations. This means you are part of the short list of applicants who will be forwarded to the supplier for their selection of the official winners.

                                          Your application will be forwarded as well as your answers to the following questions. Please answer the following questions by replying to this message:

                                          1. What is your work status? (Full time job, part time job, student, not working/retired)?

                                          I do not see how telling a third party my current work situation is relevant in any way, shape or form.

                                           

                                          I don't want to seem negative in any way, could the option "choose not to say" be added?

                                           

                                          Regards

                                           

                                          Greg

                                          • Re: Please Read: An Important RoadTest Program Change
                                            gregoryfenton

                                            That would be a great alternative, for me the answer would be "More than 10 hours" in the average week.

                                            • Re: Please Read: An Important RoadTest Program Change
                                              sarunaszx

                                              I thought that this is already being applied. I think this improvement is neccessary, and might work out fine. 13% review rate is totally unnacceptable.

                                               

                                              Also, I believe many Roadtests have applicants that are new and have no roadtest's posted before. Maybe Roadtests should include only a certain percentage of "new" members, so there are more chances of satisfied supplier?

                                               

                                              However, I tend to disagree with these short deadlines (2 weeks for unboxing for example). Roadtester's are chosen ~1month after applications are closed. Then there is shipping and etc. The time when you should reserve time for a roadtest is distant and undetermined. I simply cannot plan so much ahead and sometimes it is difficult to find spare 3 hours of free time (having a fresh head of course) to experiment / write even an unboxing review. 60 days seems reasonable to me and should be enough  for everyone to write a review (if no serious issues occur).

                                               

                                              From my perspective, RoadTest rules should encourage everyone to participate, not only the ones with the most experience / good testing equipment. That means, letting the applicant know, that the rules are strict, and he has to do the RoadTest in time, although the RoadTest itself does not have to be a masterpiece. The review from "amateurs" side can also be as insightful as from an "expert". It should be clear that the reviewer has spent time playing around, working, trying to integrate the product somewhere, but also spent some time writing the review in a clear and understandable way. For those, who are not sure how to make a roadtest, it would be good to know, what properties of the product the supplier wants to get tested / reviewed? Maybe some template could be created?

                                               

                                              Anyway, I think the changes are for the good. If we could increase the review rate, then we could expect more and better products to be supplied for testing

                                              2 of 2 people found this helpful
                                              • Re: Please Read: An Important RoadTest Program Change
                                                lucie tozer

                                                I wonder how many people apply for a roadtest and find themselves "out of their depth" when the product arrives? Sometimes I wonder if this is part of the reason many people aren't able to complete their review.

                                                2 of 2 people found this helpful
                                                  • Re: Please Read: An Important RoadTest Program Change
                                                    mcb1

                                                    find themselves "out of their depth" when the product arrives

                                                    That makes the assumption they simply apply for everything in the hope of getting something.

                                                     

                                                     

                                                    Personally I have a look at the item and decide if it is within my capabilities to evaluate it BEFORE applying.

                                                    Once the testers have been announced there is at least two weeks (assuming no delays) to download and copy links for every document available to help you.

                                                    You may also grab some piccys, and sort out software you'll need.

                                                     

                                                     

                                                    So in reality there is more than 2 months available

                                                     

                                                    Mark

                                                    3 of 3 people found this helpful
                                                      • Re: Please Read: An Important RoadTest Program Change
                                                        lucie tozer

                                                        mcb1 wrote:

                                                         

                                                        That makes the assumption they simply apply for everything in the hope of getting something.

                                                         

                                                         

                                                        Exactly, either that or just the thought of getting free stuff overshadows the fact that they've got to be able to demonstrate using it. Most of us who do contribute to roadtesting have an interest in the products but some people really are just out to get freebies. There may also be people who have dabbled a bit in electronics thinking they'll be able to just pick up how to use the product as they go along, starting with good intentions but realizing its not as easy as they thought it would be.

                                                         

                                                        Thats not to mention the Raspberry Pi craze, there was so much attention on it being the next big gadget that techies everywhere wanted one, techies in this sense people who are into gadgets (gadget show watchers etc..) and the latest crazes, lots of these people bought Raspberry Pi's but didn't have the technical knowledge to use them so they ended up been put to the back of a shelf, I imagine people like this seeing the chance of getting one for free and jump at it. It still doesn't mean they can use it when it finally arrives.

                                                         

                                                        Sorry if this seems negative but I'd prefer to think of people trying to fulfill their side of the bargain and getting out of their depth then those just taking a freebie and running with no conscience in doing so.

                                                        3 of 3 people found this helpful
                                                    • Re: Please Read: An Important RoadTest Program Change
                                                      mcb1

                                                      Interstingly looking through ACTIVITY I noted 8 people signed up for Road Test.

                                                       

                                                      Mostly all were new to the site, but one enterprising fellow had managed to amass 1325 points in a single day.

                                                      He'd started one conversation and a mention then completed the Plugged in badge.

                                                       

                                                      This made him Level3 which I guess means that he looks like he belongs and has been around.

                                                       

                                                      https://www.element14.com/community/news?streamSource=publication&streamID=821733

                                                       

                                                       

                                                      I don't know if these people all apply, but if Randall is having to sort through these types of issues, then a level based application process is not going to work.

                                                       

                                                      Mark

                                                      3 of 3 people found this helpful
                                                      • Re: Please Read: An Important RoadTest Program Change
                                                        Instructorman

                                                        Randall,

                                                         

                                                        I have been pondering the issue of dead beat road testers.  It would be great if the Road Test program could be structured around game theory principles so that a Nash Equilibrium could be achieved where all "players" take into consideration the strategies of all other "players" to reach a probability of optimal outcome.  As I see it, the Road Test program is not structured as a game that can be driven by Nash Equilibrium concepts. Here is why:

                                                         

                                                        Applicants take no risk to participate in the game and any strategy they employ in deciding their moves is not influenced by the strategies employed by others in the game.  Player strategies are effectively decoupled from each other. The participation cost is the effort required to compose and submit a road test application.  The benefit of winning is free stuff desired by the applicant.  The cost of failure to win is low, about the same as going fishing on a sunny afternoon and not catching anything.  Still had a nice afternoon outside.  Failure to submit a review after winning bears little, if any, cost to the applicant.  If an applicant is not morally invested in the community then any disapprobation by the community for failure to deliver on promises is inconsequential and therefore not a deterrent to grab and go strategies.

                                                         

                                                        As pointed out by mcb1, it is possible to masquerade as a morally invested member by exploiting point bonanzas to rapidly level up.  So, there will always be a non-zero risk in selecting unknown members for Road Test prizes.  Even your reasonable proposal to conduct due diligence screens based on past performance is not assured to prevent Road Test hit and runs. Even though, as most investment advisors will tell you, past performance is not an indicator of future performance, I think past performance is a fair indicator of likely future performance (even with stocks), so I fully support your Road Test program change because I think it will improve response rates.

                                                         

                                                        Mark A.

                                                        2 of 2 people found this helpful
                                                        • Re: Please Read: An Important RoadTest Program Change
                                                          Fred27

                                                          Wouldn't a tiered system of promotion rather than punishment be easier to manage and more difficult to game with multiple accounts, etc.? By which I mean that if you complete a good review on a less popular piece of equipment you then become eligible for the more popular items. This should increase both the quality and return rate of reviews. It's frustrating for community members to miss out on a road test you think you could do a good job on and see either no reviews appear or pointless ones. When someone is supposed to review an oscilloscope and decides pictures of their dog are relevant, you know it's just typing for the sake of typing.

                                                           

                                                          Personally, I only apply for testing something that I believe I can review well and that I have a particular interest in. I imagine most people are the same, but I'm sure there are people who just think "ooh. free stuff."

                                                          3 of 3 people found this helpful
                                                          • Re: Please Read: An Important RoadTest Program Change
                                                            reinouddelange

                                                            Hi Randall,

                                                             

                                                            I was lucky to be selected for some roadtests in the past. What I noticed, at least it is my impression, that roadtests became more and more big projects and would take too much time for me to finish. It also became a contest of who would make the most genious and biggest project out of it. (again, my impression of it) So I stopped applying because it is a hobby for me, and not my daily job.

                                                             

                                                            Will you take this into consideration too?

                                                             

                                                            I always liked the idea of the roadtest ...

                                                             

                                                            Kind regards,

                                                             

                                                            Reinoud

                                                              • Re: Please Read: An Important RoadTest Program Change
                                                                DAB

                                                                Hi Reinoud,

                                                                 

                                                                Take a look at my blog on testing.

                                                                 

                                                                Each device presented to Element 14 can be viewed in two different perspectives.

                                                                 

                                                                You use the Road Test Proposal to identify which type of testing you will do and what answers you hope to provide the vendors and the Element 14  community.

                                                                 

                                                                From my perspective, Road Testing is not about complex testing, but about assessing the device for applications within the Element 14 domain.

                                                                 

                                                                Focused testing is always useful to both the manufacturer and the community.

                                                                 

                                                                So think about what you can do with the equipment you have.  I pass on many road test opportunities as my time and test equipment is limited.  I apply for those that can provide me with a useful component for my own projects that fall within my interest and testability capabilities.

                                                                 

                                                                So be innovative, consider what you CAN do and apply.

                                                                 

                                                                Not every road test is about elaborate testing.

                                                                 

                                                                Sometimes, the vendor just wants someone to verify that the device is easy to use and works well in simple maker applications.  They need this feedback to review their own procedures and expectations of the device working in unexpected applications so that they can show potential customers the versatility of the device.

                                                                 

                                                                DAB

                                                                8 of 8 people found this helpful
                                                                • Re: Please Read: An Important RoadTest Program Change
                                                                  Jan Cumps

                                                                  A road test isn't supposed to be a contest. Don't stop applying, Reinoud.

                                                                  i have been selected a few times with a minimalist , focused entry.

                                                                  7 of 7 people found this helpful
                                                                  • Re: Please Read: An Important RoadTest Program Change
                                                                    rscasny

                                                                    Reinoud,

                                                                     

                                                                    I'm glad you posted this. I think it is helpful to take a step back and ask, What is a RoadTest?

                                                                     

                                                                    It's surely not a design challenge.

                                                                     

                                                                    A roadtest is a product review. We give you a product, you take it, conduct startup tests, conduct some experiments so you can give the reader of your review an objective opinion of the product. A roadTest is not necessarily a new product prototype thing. Of course, if you can pull it off in 60 days, that is great. I read RoadTest apps all the time that are these long manifestos of something they are going to build. And I ask myself: can someone who is a full time student or full time professional complete some long involved project in 60 evenings? I'm always leery of someone who promises the world in his or her application. No, it does not need to be a big project. You may have gotten that impression in the past, but that's not how I make recommendations. So, please continue to apply and participate in the program. Tell me something you are going to do with it that would be useful to anyone who would use the product. That's what we're looking for.

                                                                     

                                                                    Randall Scasny

                                                                    RoadTest Program Manager

                                                                    7 of 7 people found this helpful
                                                                    • Re: Please Read: An Important RoadTest Program Change
                                                                      balearicdynamics

                                                                      Hello Reinoud,

                                                                       

                                                                      your assertion is partially correct. The other members that answered you are aware of what a road test is, as you (and hopefully  me ) But when a roadtest become a gigantic project it is something out of topic, IMHO. When I enroll for a road test I always see before if I can introduce the new and almost unknown component in a project I am already working on. This simplifies the things as I see the product under the testing perspective with the advantage of using it at the same time. But we always should focus the attention on creating a report of "what the product is, what can do and what are the performances". A honest test, as a matter of fact. Just making an example, I loved a past road test where a new TI board was introduced in a real project for a car race. But it was not a project though ad hoc. I am in queue for road testing the Raspberry PI compute module 3 and I am very interested to discover as much as possible its characteristics, features and limits just because I am deciding to use it in a large industrial project. I have done the same with the LTE94112 by Infineon, this was the element to complete a project I was working by one year. So why not to test it ? A good opportunity. But a test is a test. Not a challenge.

                                                                      BTW I think that this is not the more interesting stuff that the supplier aims to read about.

                                                                       

                                                                      Look forward your future road test submissions

                                                                       

                                                                      Enrico

                                                                      1 of 1 people found this helpful
                                                                    • Re: Please Read: An Important RoadTest Program Change
                                                                      mmcphail

                                                                      it really disappoints me to read that only 13% of road tests are completed. I've applied for a few and only recently got accepted to do one so on the one hand I'm looking forward to writing a review but on the other had it annoys me that myself and probably many other people who have good ideas for a readiest miss out or don't apply because they don't feel they will be chosen. theres been a lot of interesting bits of kit on offer for review since I've signed up and it makes me wonder if people are just applying to get free stuff?

                                                                        • Re: Please Read: An Important RoadTest Program Change
                                                                          rscasny

                                                                          We are improving. But, unfortunately, there are people who register on the community, apply for the roadtest, and then cease to log in to the community for months. And a handful create mutliple accounts.

                                                                           

                                                                          Well, when I get the list of applicants, I try to select 10 or so that I think are the best to recommend to the supplier, who makes the final decision.

                                                                           

                                                                          Besides looking at the questions, I do look in to see when the applicant last logged in. If an an applicant hasn't looged in for a long time, I keep that in mind when making my recommendations to the supplier. Someone who logs in and engages in the community I feel deserves my attention over someone who is disengaged.

                                                                           

                                                                          I look for experience, but I also am looking to recommend a new roadtester. So even if you are new to the community, you still have a chance of roadtesting and receiving a free product.

                                                                           

                                                                          The final issue is your background. I am scheduled to roadtest a absence-of-voltage indicator for 600V applications. So, I would be more likely to recommend someone to the supplier who has experience at this level or perhaps facility engineering.

                                                                           

                                                                           

                                                                          Randall Scasny

                                                                          RoadTest Program Manager

                                                                          1 of 1 people found this helpful
                                                                        • Re: Please Read: An Important RoadTest Program Change
                                                                          BigG

                                                                          I just saw today's updated message and thought to respond, only to find there is a rather large chain of older responses linked to this message. So apologies if I'm repeating what someone else has said.

                                                                           

                                                                          My suggestion is that you follow the StackOverflow approach and be rigorous, and maybe a little draconian, by creating points and/or level thresholds for each road test which you then use to compare against a member's reputation points and level. The Element14 system already has a points and a level tracking system so why not put this into practice. So for example, if a new user applies for a road test and has never once contributed to a blog or helped someone, then you can treat them as an unknown entity as they have zero points and level 0 (so a high-risk candidate). Then the more road testing and contributions the more points you get / higher your level and therefore it will be easier to do the next road test etc. as they are lower risk.

                                                                           

                                                                          Now for this to work you need to be upfront in the road test description and state the minimum thresholds. Also suggest that for high-cost road test items, you raise the threshold etc. for someone to do the road test etc. This then gives you a good way to manage risk etc. of not getting a response. You could also provide "yellow cards" or even points / level deductions for those not completing the road test.

                                                                            • Re: Please Read: An Important RoadTest Program Change
                                                                              Jan Cumps

                                                                              BigG, the thing with using the point systems is that the same people (me) would get selected all the time.

                                                                               

                                                                              It's a hard thing to manage. If you give someone a yellow card, there's no stopping that person to create a new account.

                                                                              The way it's done now - selecting on the merit of the application and filtering when one hasn't logged on since submitting the application - seems to work reasonably well.

                                                                              1 of 1 people found this helpful
                                                                                • Re: Please Read: An Important RoadTest Program Change
                                                                                  BigG

                                                                                  Hi Jan,

                                                                                   

                                                                                  My suggestion is to use the points only as a minimum threshold, to help from a risk management perspective, rather than having it where the person with the highest number of points gets the product. Thus as suggested those more expensive products have higher thresholds or this is left to the manufacturer or yourself to determine. As said it is then displayed in the description... only those people with a minimum of xx points can apply.

                                                                                   

                                                                                  Deductions should apply IMHO as setting up a new account puts the person back to zero anyway which is probably much worse.

                                                                                    • Re: Please Read: An Important RoadTest Program Change
                                                                                      Andy Clark (Workshopshed)

                                                                                      The problem faced is that the points system scores community involvement not suitability for roadtest.

                                                                                       

                                                                                      The 600v roadtest is a good example where a non member with niche skills might actually be the best candidate and would sign up just to complete the roadtest. They could add value for the manufacturer and community but may not convert to a regular contributor if there is lack of stuff here to interest them.

                                                                                      2 of 2 people found this helpful
                                                                                        • Re: Please Read: An Important RoadTest Program Change
                                                                                          balearicdynamics

                                                                                          Statistics for the blog posts are able to calculate the different kind of accesses to a page; on thing is the number of views including all the people that for some reason opened a page. Another thing is the number of viewers; it is the number of people that - I suppose - really read the article or comment or interact more in depth with the post. Maybe interesting to evaluate the points giving a different weight when a user just click-and-go instead of being a viewer. Ordered by blog posts (articles, documents etc) and filtered by user name. If this can be an automated query a better design of the member can be depicted based on his participation to the community.

                                                                                           

                                                                                          Enrico

                                                                                          • Re: Please Read: An Important RoadTest Program Change
                                                                                            BigG

                                                                                            Yes that is true. But even though you may need a niche skill set for certain road tests, that person is still an unknown, with regards to the likelihood of completing a review, if they have never contributed before, and their claim to having the skills and promising to do a review is unverified at the time of application. Hence the need for some evaluation / verification tool and the use of points is at least some sort of risk evaluation. It may not be perfect but better than nothing. Then it is really up to the manufacturer, in those situations, to accept that person's application at face value, knowing that the likelihood of non compliance is quite high because of the applicant's lack of points / prior contributions. Thus you have now pushed that risk back to them.

                                                                                             

                                                                                            Remember with road tests you are required to provide one extra piece of data which can also be used for cross checking (in case of duplicate accounts) which is the physical delivery address. As teleporting has yet to materialise :-) goods have to be shipped somewhere. This probably involves more work but it does add checks and balances if warranted.

                                                                                      • Re: Please Read: An Important RoadTest Program Change
                                                                                        BigG

                                                                                        I've done some more reading through the messages.

                                                                                         

                                                                                        So it sounds like the points/level system is and has been open to abuse, but I still think this can be offset with points/level deductions and the use of yellow / red cards (of flags), which are recorded in the user's profile. So yes, IMHO you should consider the name and shame approach, whereby a failure to complete a road test is flagged on the user's profile (although, from the sounds of things these rouges simply set up a new account, but then again they then have to start from scratch).

                                                                                         

                                                                                        It also sounds like the wording used in the past to advertise road tests has created a misleading impression about it being seen as a free gift, rather than a product provided in exchange for a product review service. There may also be ways to amend T&C's so that if the person does not provide the review service then there is some sort of penalty.

                                                                                         

                                                                                        The big unknown, which I note was raised by others, is what is the person's motivation to do a road test. Maybe this shouldn't be counted against a person as there will be many reasons and maybe one reason is as good as another - yes this can put a good person off applying. I know for me, road testing is a handy approach to evaluating something new that is perhaps quite unfamiliar but you can see at first glance that it could be helpful in an existing or future project. It is often this first impression or first glance that catches people out as they find it more complex, or confusing from a documentation perspective than first envisaged. But then again that is no excuse not to complete a review, as it's really important that they still provide a report on their experiences as this hopefully helps the manufacturer improve documentation or the product itself.

                                                                                        1 of 1 people found this helpful
                                                                                      • Re: Please Read: An Important RoadTest Program Change
                                                                                        dr-jhc

                                                                                        I only found out about the RoadTest group recently, so these old posts are all new to me!

                                                                                         

                                                                                        I sometimes do contract work, and when I do I feel a large burden of responsibility to complete the job the client is paying for. A RoadTest should be no different: a professional contract "please do a review and we pay you in kind with the item being reviewed". Unfortunately, as Randall has seen there seems to be a lot of people who don't have the high standards we expect, especially when being "paid in advance" as it were.

                                                                                         

                                                                                        My company also manufactures equipment that we send out for review to professionals (i.e., established press) and bloggers (of various popularity). Similarly, we don't always get reviews done in exchange for the products we send, but it's probably more than 90% success, rather than 13% you've seen). This is probably because we are very careful to choose where we sent the products and make sure we have some good communication before the process starts.

                                                                                         

                                                                                        I'm wondering if a deposit scheme would filter out those that don't intend to do a review. Essentially, before the item is sent to you, a deposit (card payment) is taken from you, and this is repaid as soon as the review is posted. Maybe this is only for high-ticket items, or maybe the deposit amount is proportional to the item list price.

                                                                                         

                                                                                        For a high-ticket item, I'd have no problem paying a refundable deposit to show that I was serious is fulfilling the contract to which I have agreed.

                                                                                         

                                                                                        Once you've done a few RoadTests and are shown to be reliable, perhaps the deposit is no longer required.

                                                                                        5 of 5 people found this helpful
                                                                                          • Re: Please Read: An Important RoadTest Program Change
                                                                                            Instructorman

                                                                                            James,

                                                                                             

                                                                                            I see merit in your idea of requiring a deposit on high value items, or better yet, a deposit proportional to the item value ('cause some road test items have been very expensive).

                                                                                            This approach should drastically increase the fulfillment rate for reviews, but it may not impact the quality of reviews, which is another concern.

                                                                                             

                                                                                            I have read many Road Test reviews structured as undisciplined streams of consciousness.  Others come across as minimalist scribbles barely meeting the definition of a review.  Then there are reviews that are beautifully structured, well written publication grade pieces with obvious inclusion of important detail and full awareness of the needs of the reader.  The very best reviews often include high quality labelled illustrations, useful external hyperlinks, and well lit, tightly edited and clearly voiced video clips.  I prefer reading reviews written by experienced professionals that can offer expert opinions, or by hobbyists or students that are excited to share their learning experience in an engaging and thoughtful way.  Not every member has the skill to craft a well written and well structured review.  The range of skill runs from barely competent to professional grade.

                                                                                             

                                                                                            In short, there are ways to increase review fulfillment, but what are the ways to increase the consistency of review caliber?

                                                                                             

                                                                                            Mark A

                                                                                            4 of 4 people found this helpful
                                                                                              • Re: Please Read: An Important RoadTest Program Change
                                                                                                dr-jhc

                                                                                                Mark,

                                                                                                 

                                                                                                I agree that it's very important to maintain a high standard of reviews. All I can suggest is that RoadTest applicants include links to examples of what they are capable of. This doesn't necessarily need to be other reviews, but should at least show their level of competency in photo and/or video and how well they can write and structure their thoughts.

                                                                                                 

                                                                                                James

                                                                                                1 of 1 people found this helpful
                                                                                                • Re: Please Read: An Important RoadTest Program Change
                                                                                                  BigG

                                                                                                  In my opinion, I am not sure a deposit for all items will work, as it is has to be compared against the person's own opportunity cost for their time. In others words a $30 item is probably only equivalent, in value terms, to about 30 minutes of someones time. The rest of the time spent on reviewing an item is then essentially goodwill. This can then be countered by using some form of reputation evaluation or score. This is often how accreditation works. You then use this process to gain road test reputation points as a way to qualify yourself etc.

                                                                                                   

                                                                                                  Then in terms of road test reports, I think it is really important to have flexibility in style and structure, as I find when I am grappling with a problem or learning something new, in particular with software development and /or MCU based development platforms, I need to read different perspectives to understand how to do it. This, in my opinion, is why I find open source works better for me compared to corporate communication / documentation as it is much richer and varied in structure and style.

                                                                                                  1 of 1 people found this helpful
                                                                                                    • Re: Please Read: An Important RoadTest Program Change
                                                                                                      peteroakes

                                                                                                      I think over 90% of road test items would fall into the good will category, and to try to put a value on the time it takes to perform a review (My Kind of review) would far outweigh the value of the item, My first ever review was a 30$ battery booster pack from TI and E14. it resulted in over a weeks worth of work with research, video editing and schematic updates to get the review complete. even an item costing a few thousand $$ would be a week of effort to review correctly, now I will be honest and say that not all my reviews are so extensive, but in general they tend to be several 30 minute or longer videos, many pictures with words etc.

                                                                                                       

                                                                                                      The deposit I think is simply to inhibit the grab and run folks who have no interest in providing a quality review or even a review at all, they just want a freebie and run, if you have to put up even a small deposit until after review is posted then I for sure would be OK with that as I know I would always get it back. Also the minute we start to put a value on the time is the moment the customs and excise folks start to get in the way, and that is a tricky think for E14 to deal with even now. Again, I dont mind the duty on a product that is useful to me but on a product were it has no use beyond the review (The integrated mains live tester review just closed for instance) then it would be hard to justify.

                                                                                                      2 of 2 people found this helpful
                                                                                                        • Re: Please Read: An Important RoadTest Program Change
                                                                                                          BigG

                                                                                                          I like your comments. It does indeed take up a fair amount of time, but I find this is well worth the effort as it helps build up ones own knowledge base and offers an opportunity to contribute back to the community by helping others through the review.

                                                                                                           

                                                                                                          Maybe I'm being misunderstood. I was not trying find a way to value time as too problematic (as you rightly highlighted). What I was proposing was a non-financial method, by way of visible reputation points, as a way of accrediting or qualifying testers. In my opinion, this would be a simple way of achieving the objective of improving response rates. Those who do a runner would get negative points and thus are essentially user-named and shamed in the visible online reputation list (I know I proposed this earlier - would also use physical postal address as a further check to ensure new users names are not being created etc.)

                                                                                                          1 of 1 people found this helpful
                                                                                                            • Re: Please Read: An Important RoadTest Program Change
                                                                                                              peteroakes

                                                                                                              OK, that seems like a good idea but there already is a points system in place, not specifically for Road Tests but for general community participation, it has been proposed on numerous occasions to use this as a "Threshold" system to determine relative qualification for participation. Low value items would still be open to anyone of any level but perhaps need 1000 points or something, perhaps they would at least have to be a member for a certain time and earned some reputation in the community (We have had many instances were someone signs up for the purpose of a road test and are never seen again till the next road test). then more expensive items are allocated to higher level members who have proved their commitment to the community over a long period of time.

                                                                                                               

                                                                                                              I also agree, to me, this is all about giving back to the community, sharing what I have spent most of my life learning, this is why I also run a full time YouTube channel so I can share even beyond E14.

                                                                                                              2 of 2 people found this helpful
                                                                                                          • Re: Please Read: An Important RoadTest Program Change
                                                                                                            Instructorman

                                                                                                            Agreed, flexibility in style and structure is welcomed as it helps keep content fresh and interesting.

                                                                                                            As long as the writing style and structure is high quality, that is, the style and structure keep the needs of the audience in mind, then I'm all for flexibility.

                                                                                                        • Re: Please Read: An Important RoadTest Program Change
                                                                                                          Fred27

                                                                                                          I think James's idea of a deposit or similar system is a great idea. If it's simpler they maybe just get the road tester to provide payment details and sign an agreement that by a specific date they will review the item, return it or pay for it.

                                                                                                           

                                                                                                          As I write this there's one fairly recent example of a very impressive and expensive piece of equipment that has a well overdue road test. That's quite frustrating for applicants who would happily have put a significant amount of effort into a road test.

                                                                                                        • Re: Please Read: An Important RoadTest Program Change
                                                                                                          Andy Clark (Workshopshed)

                                                                                                          I was just thinking might it be a good idea to close the loop....

                                                                                                           

                                                                                                          Can we hear from the suppliers to see what it was that they liked about particular applications and reports? Perhaps that might help people get the right level?

                                                                                                          3 of 3 people found this helpful
                                                                                                            • Re: Please Read: An Important RoadTest Program Change
                                                                                                              Instructorman

                                                                                                              Yes!  I would love to hear from suppliers to better understand what it is they like to see.

                                                                                                              Every Road Tester could benefit from knowing what suppliers are looking for in a Road Test.

                                                                                                               

                                                                                                              Every supplier might have a different emphasis, and I know that each road test invitation already contains some specifics about what the supplier is looking for, but if suppliers could provide some feedback about what made them choose a particular proposal over other proposals then some of the guess work would be eliminated on the applicant end.

                                                                                                               

                                                                                                              Many of us, I believe, can adapt our road test approach to meet the specific needs of each supplier, if we know what those specific needs are when we are writing our proposals.  The down side might be that suppliers will have a more difficult time selecting the top proposals because they will all cluster around a single theme based on whatever details a supplier provides as part of the proposal criteria. There will always be room for creative variation though, so the best proposals will still stand apart.

                                                                                                               

                                                                                                              Huh, that makes the road test application process sound like a tender process.

                                                                                                               

                                                                                                              Maybe I've gone too far?

                                                                                                              1 of 1 people found this helpful
                                                                                                            • Re: Please Read: An Important RoadTest Program Change
                                                                                                              fyaocn

                                                                                                              I know it is hard for you to make such decision. But it is right for you to take some actions.

                                                                                                              In fact, self-discipline and commitment to  the promise are more import.

                                                                                                              Would it be OK if you can pull some ranking system by community members, let members do the work of evaluation for you. The ranking is feed back to road-testers public or anonymous as your choice. Let the participators eat what made of by themselves.

                                                                                                              There are many good reviews and inspire me much. I think we can mark "sucks" if it far beyond expectations.