Skip navigation

community

Up to Discussions in Medical Device Design & Equipment Development
1 2 Previous Next 1590 Views 15 Replies Latest reply: Feb 23, 2012 1:50 PM by Cabe Atwell RSS
jhjaksch Level 2
jhjaksch
15
Currently Being Moderated

Aug 16, 2011 10:33 AM

Input Device for Disabled - Sense Mouth Open

I am interested in building an input device that will provide a computer input for my daughter Liz who has Cerebral Palsy.

Liz has a great deal of involuntary head movement but she can reliably open and close her mouth to actuate a switch.

I would like to provide her with a wireless device that senses her mouth opening and closing to provide input to a single switch scanning communication system.  Years ago I combined a chin strap, a bennie cap and a micro switch to provide that function.  I am looking for ideas!

I have searched many sites looking for devices but all require her head to be held fixed which is impossible for Liz.

Attributes
  • Cabe Atwell Level 10
    Cabe Atwell
    Currently Being Moderated
    1. Aug 18, 2011 1:23 PM (in response to jhjaksch)
    Re: Input Device for Disabled - Sense Mouth Open

    John,

     

    Welcome to element14.

     

    Your question reminded me of few medical innovations.

    1. One device that reads lips, "The speechless speak again"

    2. The medical tattoo that can sense muscle and electrical impulse changes, "Tattoo electronics and possibility"

    3. An input for those who can not move at all, "The paralyzed walk, virtually, with a brain-computer interface"

     

    Since these may cost a little more than the average person will want to pay, I would start by considering the Tattoo option or some home made variant. Combine this with voice recognition, you may have something.

     

    Alternatively, you could try looking into Microsoft Kinect for computer control. Uses visual tracking and voice already. "Control the web and Windows 7 with Kinect."

     

    Cabe

  • jhjaksch Level 2
    jhjaksch
    Currently Being Moderated
    2. Aug 18, 2011 7:53 PM (in response to Cabe Atwell)
    Re: Input Device for Disabled - Sense Mouth Open

    Cabe,

    Thanks very much for taking the time to respond!

     

    The Read Lips device seems like it would  provide quite a varied array of inputs however it looks at this stage of development to be more invasive than what Liz, our daughter might be able to tolerate.

     

    The Tattoo is quite on the money.  A low power (only a meter or two distance between transmitter/receiver) Bluetooth wireless version, with lithium powered substrate layer and a thin film strain gauge might detect accurately the skin movements on her face when she opens her mouth.  I graduated from the University of Illinois with a degree in Engineering way back in 1968.  Followup in this area looks promising for Liz.  Liz is cognitively OK.  She reads, has excellent memory storage and retrieval and is currently motivated to improve her expressive language.

     

    The third item, we have tried similar BCI Type P300 devices with no success.  Getting the electrodes positioned correctly is difficult with a normal person, but with Liz almost impossible because she has a lot of involuntary head movement.  The signal to noise ratio is poor unless you find a hot spot.  Implanted electrodes are often much more effective but require surgery and can become infected.

     

    Thanks again so much for the suggestions.

  • Cabe Atwell Level 10
    Cabe Atwell
    Currently Being Moderated
    3. Aug 19, 2011 4:29 PM (in response to jhjaksch)
    Re: Input Device for Disabled - Sense Mouth Open
    I think the tattoo option, if it becomes more accurate, is the best for your case.
    The importance of computer connection for Liz is entirely up to you. If you want to try the internal EEG option there have been some advancements. Though, none of which seem at all useful at the moment.
    A sensor mesh placed on the brain is allowing people to move cursors already, "Direct contact brain sensors move cursor"
    In full brain contact is not an option, a company is looking into a EEG contact that will not get infected. "Bolt in the head reads neural signals"
    I still think a combination of facial tracking, voice, simpler touch inputs would be the easiest to implement.
    What are Liz's abilities at the moment? You mentioned jaw control. Does she have complete control? If so, track the position of the jaw in 3D space. For instance, move your jaw forward, now to the side. You could use that to move the cursor to the upper right.
    Eye tracking a possibility?
    Any other extremity have better control?
    Breathing? (breathing could suggest direction or other inputs)
    Cabe

    I think the tattoo option, if it becomes more accurate, is the best for your case.

     

    The importance of computer connection for Liz is entirely up to you. If you want to try the internal EEG option there have been some advancements. Though, none of which seem at all useful at the moment.

     

    A sensor mesh placed on the brain is allowing people to move cursors already, "Direct contact brain sensors move cursor"

     

    In full brain contact is not an option, a company is looking into a EEG contact that will not get infected. "Bolt in the head reads neural signals"

     

    I still think a combination of facial tracking, voice, simpler touch inputs would be the easiest to implement.

     

    What are Liz's abilities at the moment? You mentioned jaw control. Does she have complete control? If so, track the position of the jaw in 3D space. For instance, move your jaw forward, now to the side. You could use that to move the cursor to the upper right.

     

    Eye tracking a possibility?

     

    Any other extremity have better control?

     

    Breathing? (breathing could suggest direction or other inputs)

     

     

     

    Cabe

  • jhjaksch Level 2
    jhjaksch
    Currently Being Moderated
    4. Aug 19, 2011 8:54 PM (in response to Cabe Atwell)
    Re: Input Device for Disabled - Sense Mouth Open

    Liz is severely involved in the motor area.  She is non verbal but can make a variety of sounds. She has great difficulty initiating a controlled motor movement in response to a visual or auditory input.  The harder she focuses, the more her body freezes up.  About 25 years ago I wrote a single switch scanning program for her to communicate with educational staff as a prerequisite for attending main stream classes.  They were unable to access her cognitively without a measurement tool.  This required having Liz evaluated by therapists who observed her various hand, arm, leg, etc. movements.  We discovered she could reliably open her mouth and stick her tongue out.  This gave rise to her "yes" and "No" response to questions being structured to a binary answer.  The mouth open became the selection motor response in the single switch scanning system.  If she miss fired during scanning she would thrust her tongue to alert the aid of her motor miss.   Liz is trapped in her body and at times as she gets older becomes very frustrated.  Combining multiple motor movements for Liz would be very stressful and perhaps beyond her physical capability.

     

    We have tried the eye gaze tracking system but it requires limited head movement.  Her jaw/mouth  movements have been conditioned over many years and at this time are the direction we feel has the best chance for success.  I built a voice/sound system years ago for her and also developed a pattern recognition program using Fourier transforms to determine different facial expressions.  The microphone system was somewhat successful, but the imagining system was defeated with her constant head movements.

     

    My approach today is to allow Liz the freedom to see or hear her possible choices and respond with a sensor attached to her body/head wirelessly.  This allows her to make whatever random movements she must to respond correctly.  Hence my interest in the thin film technology on a polymer as a possible course.  It could be portable and noninvasive (no bolts in the head).

     

    The ideal augmentative communication system for Liz would collect brain signals that are her thoughts and convert them to language.  Man/machine interfaces are improving ever so slowly in this area.  We hope the gap will shrink.  We never give up!

     

    Thanks again!

    John

  • Cabe Atwell Level 10
    Cabe Atwell
    Currently Being Moderated
    5. Aug 24, 2011 9:44 PM (in response to jhjaksch)
    Re: Input Device for Disabled - Sense Mouth Open

    Upgrading your previous jaw/mouth button and tongue sensor with modern fabricating methods is a great stopgap solution until something better comes along.

     

    How do you envision the sensor working? How do you see it attach?

     

    Here are my thoughts: I would start by trying to build the device into the mold of a dental guard. Like what some people wear a night to avoid grinding of the teeth.

    What I would like to try is molding all the electronics needed into the guard itself. A handfull of buttons molded inside the guard itself, along with a microcontroller, wireless transmitter, battery, and a coil to use for induction charging.

     

    All these parts are off the shelf, near plug and play.

    The uC immediately sends the data (which button) on every press. Simple as that.

     

    If you are willing to give it a try, I (and hopefully the community) will help in this effort.

     

    Cabe

  • timatthelab Level 2
    timatthelab
    Currently Being Moderated
    6. Sep 11, 2011 6:19 PM (in response to jhjaksch)
    Re: Input Device for Disabled - Sense Mouth Open

    I'm sorry for the late reply.

     

    If you have already accepted accessories (chin strap, cap), would a pair of accelerometers work? The differential signal would indicate an opening or closing of the mouth. A nod of the head would be ignored without a difference. Some filtering would be required.

     

    Both signals could be fed into a low power microcontroller and a low power radio, running from a coin battery or some small rechargeable.

     

    The reason I mention this option is to allow you a choice of avoiding a long-term (even if only used sometimes) contact with any kind of electrode. There are not many materials, particularly conductive ones, that you can leave in long term contact with the skin without some kind of adverse affect.

     

    Tim

  • jhjaksch Level 2
    jhjaksch
    Currently Being Moderated
    7. Dec 14, 2011 8:44 AM (in response to Cabe Atwell)
    Re: Input Device for Disabled - Sense Mouth Open

    Have started building a device and I am finding the wireless transmitter, battery, and a coil to use for induction charging are all too physically large. Is there a package that is the size of a microcircuit or smaller?  I am looking for wireless communication. I don't need a battery.  Just need to interrogate the sensor(s) to get the data.  What are the options with battery and no battery?

    Thanks much

    John

  • YT2095 Level 7
    YT2095
    Currently Being Moderated
    8. Dec 14, 2011 9:11 AM (in response to jhjaksch)
    Re: Input Device for Disabled - Sense Mouth Open

    this is not in Any Way meant as an Answer to your/her challenge, just a few ideas.

    I was thinking perhaps something along the lines of an RFID type stick-on tag that would require no power but would activate a Theramin type of receiver, also can she control her breathing? perhaps a small blow-pipe to activate or puncuate or even comunicate using?

    I`m also thinking something along the lines of a Kinect type of non intrusive/intimate contact reader as well, I have no idea How one would code this, and all would certainly need discriminator type filtering for random movements.

    all the best and Gods blessings to you both

     

    Message was edited by: YT 2095 to add: can you say a little bit more about the Type of modulation of her communication in these movements, does she have Fine controllable skills on top of Large movements or is it Large (Macro) Movements overlaying finer random movements, this May help us elicit cogent data/communications from her and sensor types.

  • Cabe Atwell Level 10
    Cabe Atwell
    Currently Being Moderated
    9. Dec 14, 2011 3:12 PM (in response to jhjaksch)
    Re: Input Device for Disabled - Sense Mouth Open

    John,

     

    I am glad to see you are still working on the idea.

     

    I looked around at various wireless modules. My goal was to find something that is as all-in-one as possible. I think Zigbee might be the way to go. The communiy surrounding Zigbee will sample code and schematics for connection to any type of processor you choose. Let me know what processor/microcontroller you want to use, and I will see if I can dig up some sample work for it.

     

    You could try the Zigbee system on a chip. The QFN-40 package is fairly small. With Zigbee you can string more modules around to ensure the data connection.

     

    Cabe

    http://twitter.com/Cabe_e14

  • RWM Level 6
    RWM
    Currently Being Moderated
    10. Dec 14, 2011 4:10 PM (in response to Cabe Atwell)
    Re: Input Device for Disabled - Sense Mouth Open

    As human body has higher temperature inside, perhaps it is possible to detect mouth opening remotely using thermal camera or array of thermopiles?

     

    Using Kinect I think is also promising. There is a "Kinect Accelerator" program with technical help from MS, but for companies only:

    http://www.element14.com/community/community/news/blog/2011/11/30/microsoft-kinect-contest--and-the-3d-topographical-display

  • jhjaksch Level 2
    jhjaksch
    Currently Being Moderated
    11. Dec 14, 2011 9:10 PM (in response to YT2095)
    Re: Input Device for Disabled - Sense Mouth Open

    The Kinect approach would utilize facial recognition which I tried 18 years ago.  I built a system using a video camera, frame grabber, image processing with eigenface algo.  It worked if Liz's head was completely still.  Today she can not tolerate her head immobilized.  A second problem is the ambient lighting and getting a frontal image.  The state of the act has certainly improved with huge increases in processing power but the algos are still lacking in reliability.  RFID is a good way to go. No power.  Interrogate the  sensor(s).  Another possibility is to energy harvest from the surrounding area to provide power for the circuits.  Having the sensor directly on her face is the best data collecting site.  The epidermal electronic patch needs to be completely integrated containing either a power source and transmitter or passive RFID type. 

  • jhjaksch Level 2
    jhjaksch
    Currently Being Moderated
    12. Dec 14, 2011 9:30 PM (in response to Cabe Atwell)
    Re: Input Device for Disabled - Sense Mouth Open

    Cabe,

    Good to hear from you.  Zigbee looks good for a protocol or possibly IEEE 802.15.4.  RFID is a good way to go. No power.  Interrogate the  sensor(s).  Another possibility is to energy harvest from the surrounding area to provide power for the circuits.  Having the sensor directly on her face is the best data collecting site.  The epidermal electronic patch needs to be completely integrated containing either a power source and transmitter or passive RFID type.  They are getting close to a prototype with more integration.  The patch is non invasive, extremely thin, can measure strain, temp and area potentials.  The current prototypes need power.  They are just raw analog signal circuits.  Connecting wires to the patch is lab work.  Finding a micro sized package that could be placed on her cheek, attached to the patch circuit is not something I am finding.  I would take apart, just to test the patch circuit, a micro miniature, wireless sensor of almost any type just to get the back end processing.  Or if I could find a RFID package, again micro miniature that could be wired to the patch and the whole thing covered with some 3M Tagaderm.  Its a tall order but the other option is to wait a while longer until the EE patch gets more integrated.

    John

  • jhjaksch Level 2
    jhjaksch
    Currently Being Moderated
    13. Dec 14, 2011 9:33 PM (in response to RWM)
    Re: Input Device for Disabled - Sense Mouth Open

    I am trying to sense directly on Liz's body.  Camera not as good.

  • Cabe Atwell Level 10
    Cabe Atwell
    Currently Being Moderated
    14. Dec 14, 2011 10:20 PM (in response to jhjaksch)
    Re: Input Device for Disabled - Sense Mouth Open

    John,

     

    Aside from being great for mobile identification, RFID for a power source from sensors and microcontrollers (uC) is still on the fringe of research. If you can break that field, by all means.

     

    If you are having issues, then external battery power may be the only solution.

    Place the battery into the patchwork. With low power components, you might be able to do a lot.

     

    Many of the links I would like to show you are down on element14. What I would show is just examples of how doing what you propose is possible using standard materials and tech.

    A recent development might not show jaw movement, but it can accomplish everything else you want. See the pajama that is a complete baby monitoring system. This could be attached to the body and give you all the data you mentioned.

     

    However, only sensing jaw movement will make the patch simple.

     

    Something has to sense particular jaw muscle signal firing. From here data could be sent via wires to a body harness, like the pajama system above. You would have the best of all worlds, an off-the-shelf monitoring system with your cheek patch communication system.

     

    To go wireless, you will have to integrate a transmitter and battery system in the patch. Finding an off-the-shelf  product to do all this is going to be impossible. I suggest we start by finding the lowest power transmitter possible and a small form factor uC to pair with it.

     

    Cabe

    http://twitter.com/Cabe_e14

1 2 Previous Next

Related Content


Related Products
Discussions
  • Retrieving data ...

Incoming Links

Bookmarked By (0)

Legend

  • Correct Answers - 50 points
  • Helpful Answers - 25 points