17 Replies Latest reply on Nov 13, 2019 8:33 AM by rsc

    Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work

    rscasny

      It's been about a year since I last had a conversation with the Roadtest Group. This will be be my fourth conversation, where I share with you RoadTest results, policy changes, my thoughts on improvements, and anything else that I feel is important or what RoadTest group should know about. So, let me get to it!

       

      2019 Results to Date

      I went back and looked at the completed Roadtest list for 2019 and recorded the completion percentage for each one. This list excludes those roadtests in enrollment or recent ones that are now in the reviewing period. Here are the results:

       

      • 10 roadtests - 100% compliance
      • 10 roadtests - 75-99% compliance
      • 5 roadtests - 50-74% compliance
      • 2 roadtests - less than 50% compliance

       

      Pretty good overall! I'd like to thank my roadtester superstars. These are great results. We are in a completely different place than were we were two years ago when I took over managing the RoadTest program. Back then, my first roadtest audit revealed only a 13% completion compliance.

       

      I realize many of the roadtesters have full time jobs with families, so your time is limited. So, these results reveal a high level of commitment and professionalism. Your participation is truly appreciated. I can't thank you enough. It is your work that makes the RoadTest program interesting and thriving on the element14 community.

       

      If you haven't completed your roadtest, I encourage you to post it on element14 as soon as you can. Thanks.

       

      My Philosophy: What Is a Roadtest?

      Before I get into some of the RoadTest program news and changes, I wanted to share with you my philosophy on Roadtests. I have mentioned it a number of times to members, sometimes within my comments right on RoadTest pages. But I want to share it with the group again. So, what is a RoadTest? Perhaps it is easier to say what it isn't. I don't think a RoadTest is always a project build. Given the limited time of a RoadTest, there may not be enough time to build a project or design a prototype. Of course, this would depend on the product and the RoadTester. I don't think a RoadTest is a means to obtain a free product to test out in a live system or mission critical application (more on this later).

       

      What is a RoadTest? A roadtest is simply an experiment with a product where the tester gets to know it well enough to see how it responds or spot operational issues or problems with the documentation. Of course, a roadtest depends on what a tester wants it to be given the limited time I give for a RoadTest. A RoadTest review should give the reader an experience of the product such that he would be interested in exploring or working with the product himself or herself. Of course, in the end, what a RoadTest is depends on what a RoadTester wants it to be.

       

      Re-Considering the 60-Day Roadtest

      I think the length that I give to conduct a RoadTest (60 days from the receipt of the product) has been debated for a long time. Some people think it should be longer. Others think it's fine. And still others look at the 60 days in terms of their own busy professional and personal lives--sometimes it's too short when they are busy and sometimes too long when they have nothing to do.

       

      Why do I give 60 days? Why not longer? I recall a couple of roadtests where there were some problems and the RoadTesters were going back and forth with the supplier's tech support. This is a special situation and of course the 60-day time limit should be suspended in these situations and the RoadTesters given more time. If something does come up during the course of a RoadTest (a problem with a dev board or a personal health issue), communicate it to me and I will adjust the due date accordingly. Again, these are special situations.

       

      Why am I reluctant to extend the standard 60-day time frame to 70 or 80 days or more for most RoadTests. I can answer that question with certainty: my data shows that if someone has not completed a RoadTest in 60 days, and has not made an effort to communicate to me about a problem (or even respond to my assistant when he sends out follow up emails), the RoadTester will not complete his review at all. What I think happens is that beyond 60 days, peoples' lives change in unpredictable ways. Things come up that have greater priorities and the roadtest is viewed in a rearview mirror. I can appreciate that . I don't know what will come up in my life in three months down the road. But as a policy goes, 60 days was chosen because it's enough time to experiment with the product yet not so long that if some has some life change down the road it will not prevent them from completing the RoadTest. Having said all this, I think there are some situations that would require a long RoadTest; if that's the case, communicate it to me and we can work it out. I encourage you to offer your thoughts in the comments section.

       

      What Happens When a RoadTest Is Used To Deploy a Solution at Work or a Customer's Site

      I frequently receive RoadTest applications that propose to use the RoadTest product in a prototype, a mission-critical system, a research project or a machine/control at a manufacturing facility or a laboratory. And this is great; I want our engineers to consider our products in their designs. But what often happens is that the RoadTester has to wait on the client to approve something, or a third party needs to do something before the RoadTest can begin, or even there often is a change in plans and the RoadTest is replaced by other things with greater priority. The question arises from where I sit: is this something that is within the scope of a RoadTest? I would rather help you connect to people in our company who deal with this sort of implementation and/or testing. A RoadTest is 60 days long. You really have only about 45 days to actually do the testing. Not much time in a real production situation. If you are a professional who wants to use a RoadTest product, contact me directly, I will reach out the the people here to assist you.

       

      New Policy on RoadTest Product Returns

      If a RoadTester has not completed or decides not to complete a RoadTest review, I felt that the product should be returned so I can send it to another person who applied. This policy has caused some inconvenience and some complaints. I took this problem to my supervisor. After have a long discussion about it with my supervisor, I will not be asking that all products will be returned. We will probably establish a minimum threshold for return in the $100-200 range and up. I think we came to the conclusion that a returns policy is not the issue that needs to be addressed; rather, we need to modify our application screening process such that the people I choose do indeed complete their reviews. Expect to see a small modification to the application going forward.

       

      The Student's RoadTest

      We also had a long discussion about how to get more students (first or second year undergraduate) involved in roadtests. We realize that students are still learning about electronics and may not have the background to do a RoadTest that an experienced engineer may write. And that's fine. What I will be rolling out in the next few months is the Student's RoadTest with clear expectations what this special group needs to do in a RoadTest.

       

      Policy Reminder: Multiple Accounts Don't Count

      If I haven't explicitly clarified it before, let me do it now. If someone is applying for a RoadTest under two or three different accounts, both applications will be disqualified.

       

      New Emphasis on Community Participation in Selecting RoadTesters

      While we are getting reviews done at a better rate than in the past, there are still a considerable number of reviews that never are written. I audit the roadtests regularly and it is clear to me that a good number of people who do not write reviews do not participate on the community. If you are engaged (e.g., write a blog, make a comment, take a quiz, watch a webinar) you are more likely to write your review. So, I will be re-emphasizing community participation in selecting roadtesters. Watch for changes in the roadtest application.

       

      Don't Know Enough to Be a RoadTester--Not!

      I recently ran a poll/discussion on why a member is not more active in the RoadTest Program: Poll: How Can I Get You More Active In The RoadTest Program? One of the things that came up is that some people feel they are not as knowledgeable as they need to be to perform a RoadTest. But isn't that the point of a RoadTest? You take the challenge of doing a RoadTest to learn. Yes, you need to understand the basic idea of the product and you need to be able to comprehend the docs. But then go at it and ask questions. That is why there is a comments section on the RoadTest page. Ask the community questions, and I'm sure they will help you. If you feel you need a mentor, I will ask experienced members if they would have a bit of time to provide special help. So, reach out to me. Your questions can help all the people who are reading the comments also seeking help. Last thing: suggest some products you feel you know enough to roadtest. I will see if I can put them on the schedule.

       

      Sincerely,

       

      Randall Scasny

      RoadTest Program Manager

        • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
          colporteur

          Great summary.

           

          While reading, my thoughts drifted to a mentor-ship program. There are some really brilliant minds visiting and sharing on this site. Getting a chance to partner with them to complete a RoadTest, would benefit them and our community.

           

          Sean

          • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
            ntewinkel

            That all sounds good, Randall,

            Looks like a good way to handle what is generally a tricky thing to manage.

             

            > ...it is clear to me that a good number of people who do not write reviews do not participate on the community. ... I will be re-emphasizing community participation in selecting roadtesters.

            That sounds like a really good idea.

             

            ps, what does a 99% compliance mean? or partial compliance in general.

             

            Cheers,

            -Nico

            • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
              jissdeodates

              rscasny, just a small thought on the student roadtests. All students do projects in their college final year. Particularly for those with electronics and computer science background, roadtests are a great chance to have the motivation to research more so as to effectively learn and do by their own. Of course, the guidance from the specialists at element14 will be an added value.

              8 of 8 people found this helpful
              • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                Fred27

                All good points. I can see the road test program is going from strength to strength recently. It looks like it will continue improving.

                • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                  lui_gough

                  I think I probably talk too much around the RoadTest area already as it is - so I'll try to keep it as brief and relevant as I can.

                  rscasny  wrote:

                  Here are the results:
                  • 10 roadtests - 100% compliance
                  • 10 roadtests - 75-99% compliance
                  • 5 roadtests - 50-74% compliance
                  • 2 roadtests - less than 50% compliance

                  Compliance rates look much better than before, but I think there should be an emphasis on not only compliance but quality and/or engagement. I don't say this only because I feel like I put in a decent effort into my reviews, but ultimately the sponsor and community doesn't benefit from completed three-paragraph reviews that might otherwise tick the "compliance" box. Previously, we may have mentioned supplier feedback or involvement post-review - I understand some suppliers may be a little hesitant to be too involved, but I think it's probably something to think about.

                   

                  rscasny  wrote:

                  So, what is a RoadTest? Perhaps it is easier to say what it isn't. I don't think a RoadTest is always a project build. Given the limited time of a RoadTest, there may not be enough time to build a project or design a prototype. Of course, this would depend on the product and the RoadTester. I don't think a RoadTest is a means to obtain a free product to test out in a live system or mission critical application (more on this later).

                   

                  What is a RoadTest? A roadtest is simply an experiment with a product where the tester gets to know it well enough to see how it responds or spot operational issues or problems with the documentation. Of course, a roadtest depends on what a tester wants it to be given the limited time I give for a RoadTest. A RoadTest review should give the reader an experience of the product such that he would be interested in exploring or working with the product himself or herself. Of course, in the end, what a RoadTest is depends on what a RoadTester wants it to be.

                  Strongly agree with the open ended nature of the RoadTest - I always try to leave some room to pursue features and quirks which may not be apparent from the documentation. That being said, documentation review can easily be done prior to product being received, but somehow, I'm not sure all reviewers take the time to familiarise themselves with it at an early (i.e. application) stage sufficiently to plan their applications, hence the number of vague "I'll figure it out when I get it" responses. Perhaps less emphasis on "free" and more on "this is your chance to get your hands on X and answer those burning questions you have that aren't answered on paper/get hands on experience". A different kind of sell, and perhaps one we've vaguely bought up before.

                   

                  rscasny  wrote:

                  I think the length that I give to conduct a RoadTest (60 days from the receipt of the product) has been debated for a long time. Some people think it should be longer. Others think it's fine. And still others look at the 60 days in terms of their own busy professional and personal lives--sometimes it's too short when they are busy and sometimes too long when they have nothing to do.

                  I encourage you to offer your thoughts in the comments section.

                  Personally, I think the 60-day period strikes a happy medium and should be retained. I've done reviews outside of e14 on much shorter timeframes - often this is not sufficient to get a good grasp of the product and understand potential durability or deep functionality issues. Beyond 60-days is a tough-sell to some sponsors, especially when their product launches are perhaps strategically timed and they would like content by a certain date. I think some people would like longer to do further exploration, but having nothing delivered in the interim is a big risk and whether they can maintain the motivation in the longer term is a question. Nothing better than a hard deadline to motivate a RoadTester, but also, it's important for the RoadTesters to also have some closure. Once they see the deadline and have delivered, they can "decompress" and feel relieved that they have met their brief as best as they can - otherwise they could be "on the hook" for way too long and never get anything else done on the weekends. I think community expectations, especially if the period is long, could be disastrous as it's not a community RoadTest (per-se) but a RoadTest done by an individual, and I think perhaps some of us (including myself) may have been guilty of expecting too much from others without considering their expectations. It may open up the potential for too much back-seat driving.

                   

                  rscasny  wrote:

                  I frequently receive RoadTest applications that propose to use the RoadTest product in a prototype, a mission-critical system, a research project or a machine/control at a manufacturing facility or a laboratory.

                  Agreed - that's not a great idea. Having to deal with the manufacturers already can be a bit of a back-and-forth, but adding more external dependencies will likely cause issues. Some niche products are hard to RoadTest without access to a particular set-up, but I think such RoadTest propositions probably need a Plan B that runs simultaneously side-by-side, or have the "deployment" as a stretch goal rather than the main RoadTest.

                   

                  rscasny  wrote:

                  If a RoadTester has not completed or decides not to complete a RoadTest review, I felt that the product should be returned so I can send it to another person who applied. This policy has caused some inconvenience and some complaints.

                  I can understand the complaints - often postage from AU back to US is going to cost more than the product, especially if it's heavy. Ultimately, I thought it would be an added incentive to complete the review and not be out of pocket ... but it seems others may have been inconvenienced because they took this in the wrong spirit. Perhaps this may lead to a new "incentive" scheme to review low-value items rather than a "penalty" for not delivering.

                   

                  rscasny  wrote:

                  We also had a long discussion about how to get more students (first or second year undergraduate) involved in roadtests. We realize that students are still learning about electronics and may not have the background to do a RoadTest that an experienced engineer may write. And that's fine. What I will be rolling out in the next few months is the Student's RoadTest with clear expectations what this special group needs to do in a RoadTest.

                  That's an interesting initiative and one I would definitely support as I did try to tell my colleagues and fellow peers when I was still a student to join the program and sharpen their skills while earning some gear. Very few of them succeeded in winning any applications at the time - would be good but perhaps there needs to be clear eligibility criteria to stop us "more experienced" non-students from competing? Perhaps we need to show our student cards and use an edu address .

                   

                  rscasny  wrote:

                  If I haven't explicitly clarified it before, let me do it now. If someone is applying for a RoadTest under two or three different accounts, both applications will be disqualified.

                  Seems fair to me. I never thought anyone would have enough time to submit an RT application under several accounts ... but that's definitely one way not to win.

                   

                  rscasny  wrote:

                  New Emphasis on Community Participation in Selecting RoadTesters

                  While we are getting reviews done at a better rate than in the past, there are still a considerable number of reviews that never are written. I audit the roadtests regularly and it is clear to me that a good number of people who do not write reviews do not participate on the community. If you are engaged (e.g., write a blog, make a comment, take a quiz, watch a webinar) you are more likely to write your review. So, I will be re-emphasizing community participation in selecting roadtesters. Watch for changes in the roadtest application.

                  When I read the title, I almost thought you meant that there will be an emphasis on the community in selecting RoadTesters which would make things rather interesting. But instead, I realise what you meant is a greater weighting on community participation. I will wait and see what this means with regards to how a RoadTest application looks like.

                   

                  rscasny  wrote:

                  One of the things that came up is that some people feel they are not as knowledgeable as they need to be to perform a RoadTest. But isn't that the point of a RoadTest? You take the challenge of doing a RoadTest to learn. Yes, you need to understand the basic idea of the product and you need to be able to comprehend the docs. But then go at it and ask questions.

                  I think this is a double-edged sword and really depends somewhat on the product. Of course, if the member is willing to learn, the RoadTest process can be highly beneficial for them and they can develop insights from the perspective of a novice. However, other products may demand reviews from a higher technical level to appeal to those who might be in the market for the product. I think the wisest applicants are those who know when they are biting off more than they can chew (with the two months of time given, their existing skill-base and time available) and perhaps elect to let a RoadTest pass by so that they won't be the one posting a review that says it was "too difficult to get working". Ultimately, the applicant who does not apply has no chance of winning, but they in turn may avoid banging their head against a wall or turning in an application which costs time to be judged but would not be competitive anyway.

                   

                  Of course, this is just my 2c from my perspective, as sometimes you need to be very careful of what you wish for (again, myself included).

                   

                  - Gough

                  8 of 8 people found this helpful
                  • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                    stevesmythe

                    It's good to see that the "compliance rates" have improved so much under Randall's guiding hand.

                     

                    I know it's subjective, but I agree that the "quality" of the RoadTest report is also important. That doesn't mean it has to be several thousand words (which can be a bit off-putting), but I think a RoadTester should demonstrate that they have made an effort commensurate with the device they are testing.

                    5 of 5 people found this helpful
                    • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                      three-phase

                      Whilst I agree that improving the quality of roadtest is important, you have to walk before you can run, so the improvement in the return rate of the roadtest is great to see.

                       

                      Improving quality is also harder than improving the response rate and ultimately requires feedback. I wonder if this is something like providing mentors to those completing a roadtest, if they wanted one, could help in improving quality and also not be too onerous for element14, if some of the more experienced roadtesters would agree to assist with the mentoring?

                       

                      Kind regards.

                      4 of 4 people found this helpful
                      • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                        andrewj

                        Interesting post.  From my perspective, I feel like too much of a novice and have ruled myself out of all those that have come up since I joined, but I never considered it from the perspective you give Randall.  I find that really encouraging and I’m generally pretty good at figuring stuff out to a reasonable level so maybe I’ll have a longer think about future ones.

                        • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                          rscasny

                          I read a few comments about the quality of roadtest reviews.

                           

                          Needless to say, quality is important. But the challenge since I took over management of the roadtest program has been choosing the right people who will write their reviews. Thus, I have been focused on understanding the profile of a roadtester who completes his/her review. While knowledge is important, a recent login and community participation level are more indicative to people who indeed write their reviews. I have to report back to the sponsor and while they understand there maybe extenuating circumstances for not completing a reviews (i.e., illness), they are participating in the program because they want to get feedback on their products.

                           

                          I feel the quality of roadtest reviews varies. I have observed that people who participate a loft in the community write more detailed, quality reviews. New members sometimes write high quality reviews, sometimes not. And I think, in their defense, they may not understand the expectations. So, I have to do a better job on-boarding new members of what we are looking for.

                           

                          But, honestly, for the most part I think the quality of the reviews is pretty good. I can only think of a few that I felt a reviewer wrote a review that made me think, he is just putting words down to say he did it. I do privately give feedback to these reviews and urge them to take another look. What I do not do, nor have ever done, is to tell people what to write. I will give them suggestions on writing style and communication, but never content choice or opinion. We want honest feedback. If the part has a flaw, the sponsor wants to know; they want to improve and respond to you, their customers.

                           

                          Randall Scasny

                          RoadTest Program Manager

                          6 of 6 people found this helpful
                          • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                            BigG

                            We all get questions post road test report from other members, which are usually very good questions, but I sometimes find that these questions would've been outside my road test plan and hence would never have been addressed.

                             

                            As such, I am wondering whether there was some way of capturing members open questions before a road test, then if these were somehow formalised in some way, then a road tester will know in advance what others (i.e. the audience) are wanting to know and hopefully then, as part road test plan and report, these can then be addressed. This should then reduce miss opportunities to get answers to those burning questions and handle all those unmet expectations.

                             

                            Anyway, it's something to think about as needs a bit of thought so that it does not overload a road test with questions for the sake of it.

                            7 of 7 people found this helpful
                            • Re: Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work
                              rsc

                              I have tried many times to get students and other instructors to join element14 with little luck.  The professors think it's a kids' thing and the students won't join anything where they can be monitored by adults.

                              I only know of three individuals that have signed up from my suggestions.  I've also tried to get together teams for design challenges, but everyone's "too busy".

                              Scott

                              2 of 2 people found this helpful