Also on YouKu, for those in China: http://v.youku.com/v_show/id_XMjY4Mzc4ODQ0.html
Load Google Translate
XinCheJian wrapping up: technical knowledge acquired and lessons learned
Posted by rngadam May 3, 2011
我很高兴能参与到这个教育机器人项目里面来！我负责项目里面机器人脸部设计与控制部分。
一开始想了很多，想要机器人可以在老师视频和机器人的脸可以互相切换展现在学生面前。机器人还能给学生出简单数学题目 或玩一些简单益智游戏。
因为我比较熟悉的是FLASH技术，我还曾经想FLASH通过RED5进行机器人视频通信，还有设计一个更美观的控制端界面。但无奈时间太少了。
以下就是当时设计的脸部的雏形，有点呆。到时候我希望能控制机器人的眼睛、嘴巴、眉毛等部位。甚至我设想还能让它做一些类似动画片里稀奇古怪的表情，甚至鼻子喷气，会害羞脸变红等等。
目前已经可以通过过android apk程序嵌入webkit --> HTML(javascrip)控制 --> 机器人的脸部(FLASH)。
提供简单一些控制：双眼运动、单眼运动、瞳孔大小、观察角度、嘴巴张合次数。
这是控制端的截图：
Min Lin Hsieh 谢旻琳: our community manager, director of purchasing, president of marketing, master of social media relations, etc...
Danzel Lee 李劲, Swat Su 苏迪, Worlf Wang 王宇: photographer, (future) mechanical engineer, woodworker
Michael Liao: amateur mechanical engineer
Maksim 林秋: UI design, interaction expert
Lionello Lunesu: hacker!
Lionello Lunesu: daytime software developer and nightime hacker!
...and Ricky Ng-Adam 伍思力, project generator and software development
The clock read T-14 hours and keeps on ticking [http://www.timeanddate.com/counters/customcounter.html?msg=GGHC+Deadline&month=5&day=4&year=2011&hour=0&min=0&sec=0&p0=224] still, but the problem is that in Shanghai it's already past midnight on Wednesday leaving precious little time to catch up on some sleep and get back to work on the final stretch.
Today was a real marathon with a big chunk of code written and a few bugs quashed (with a few more added no doubt!).
Lionello was a big help: he hotglued the loose screws in the servo shaft to make them more reliable and installed Android 2.3 in the Haipad m701-R. He was also pleasantly surprise at how much hardware we get in such a compact package for such a small price.
All our software is in our GitHub:
It's very messy admittedly. What we plan to actually use for the demo:
Main app is this (server app, Haipad side)
The Main app serves the client as HTML + flash to the client webbrowser
Particularly this HTML with Javascript (where I need to put the Javascript + SWF)
This is the Arduino controller part
If people have example code, I'd appreciate it.
The clock reads T-14 hours and keeps on ticking but the problem is that in Shanghai it's already past midnight (Wednesday early morning) leaving precious little time to catch up on some sleep and get back to work on the final stretch.
Today was a real marathon with a big chunk of code written and a few bugs quashed (with a few more added no doubt!).
Lionello was a big help: he hotglued the loose screws in the servo shaft to make them more reliable. He also installed Android 2.3 on the Haipad m701-R.
He was also pleasantly surprise at how much hardware we get in such a compact package for such a small price!
All our software is in our GitHub:
https://github.com/rngadam/XinCheJian-GGHC/
It's very messy admittedly. What we plan to actually use for the demo:
- Main app is this (server app, Haipad side)
- The Main app serves the client as HTML + flash to the client webbrowser
- Particularly this HTML with Javascript (where I need to put the Javascript + SWF)
- This is the Arduino controller part
XinCheJian demo'ing Arduino servo control from Android using Bluetooth
Posted by rngadam May 1, 2011
Yes, two weeks since the last updates. You guys must have been wondering where we were?!
As it is, we were heads down spending oodles and oodles of time researching the various possible solutions to stream video. Lots and lots of time. Hours and hours. Actually every waking hour for two weeks Enough to suspend every other life activity (including cancelling Chinese classes for this week!).
Without video streaming, there's little in way of tele-presence... But the challenges of video streaming makes every little other problem in our project trivial.
On the Android platform, there's little in way of built-in solutions, especially solutions that would let us stream and be OpenSource so we could integrate it into our project. Found IP Camera For Android [http://code.google.com/p/ipcamera-for-android/], a project that promised h.264 streaming out of the camera. However it didn't work with our platform for reasons that were at that time quite nebulous.
So we temporarily gave up on "IP Camera for Android".
We then spent a larger chunk of time researching how to get ffmpeg to work on Android which required getting familiar with cross-compilation to the ARM platform. The result, sadly, even after exploring all avenues we could think of to access the low-level video4linux interface (/dev/video0) on the Haipad was just random noise.
So we then went back to IP Camera for Android and really understand how it worked so we could make the simple required fix...
Fundamentally, to understand the issue with the output generated by the Android encoder we have to understand that a video file is two things: the encoded stream (using a codec, in this case H.264 AVC) and the "container" of that content that can be decoded can contains information to describe the video stream. In the case of the MP4 container format, that description comes into the form of "atoms" or "boxes" of information.
The problem is that the output produced by the Android encoder is not designed for streaming; this means that those boxes are not written until the stream has been fully written out to disk. This also means that the header information that would let a codec parse the output is not available until the end; in the case of H.264 the quite essential information we want is the SPS (Sequence Parameter Set) and PSS (Picture Parameter Set (PPS)).
The nice hack that "IP Camera for Android" uses is to create a sample video on disk and reads the SPS and PSS information from that. Since that output between encoded videos at the same resolution coming from the same encoder is consistent, that same information can be reused for other video streams.
For streaming, it captures the video then inserts a "StreamingKernel" between the encoder output (which luckily we can redirect to a local socket), skips the incomplete MP4 header, inserts the previously read SPS and PSS and constructs a more streamable VLC stream, another video container format. The VLC stream is made up of the individual H.264 frames wrapped in VLC "tags" served from an embedded WebServer 
to a flash based video player (!!) on the client side.
to a flash based video player (!!) on the client side. Yes, this means your phone can run a webserver that serves real-time encoded video! When you think about that, it's kind of pretty awesome to realize that we live in the future...
The problem was that the software made some simple but erroneous assumptions about the format of both the sample video and the length of the MP4 header. Actual encoded video data starts after an "atom/box" called "mdat". The mdat actual position varies depending on what the encoder is producing - the original code set the header size is 32 bytes while post-2.1 devices are 44 bytes.
As fixing the size of that header is annoying, we modified the code to read byte per byte from the stream until we get to mdat and then actually doing the streaming. In addition, we've made changes to reduce the number of costly data copies and temporary buffers to "streamline the streaming".
The result is still not entirely satisfactory due to the latency that seems to be introduced by buffering inside the encoder... But it's a big step forward.
This is quite a detour into the depths of Android and streaming. It did give us valuable experience with the platform and we can now use that experience as a building block for other interesting projects. But it also came at the cost of lots of time and energy that could have been used making progress elsewhere in the competition project...
So now we have to pick all the pieces we've done in one coherent whole...
You can follow or find more details on Github [https://github.com/rngadam/XinCheJian-GGHC] and the wiki [https://github.com/rngadam/XinCheJian-GGHC/wiki] as we try to put all the pieces together into a demoable final project. More updates on other aspects of the project soon...
Yes, two weeks without updates. You guys must have been wondering where we were?!
As it is, we were heads down spending oodles and oodles of time researching the various possible solutions to stream video. Lots and lots of time. Hours and hours. Actually every waking hour for two weeks... Enough to suspend every other life activity (including cancelling Chinese classes for this week!).
Without video streaming, there's little in way of tele-presence... But the challenges of video streaming make every little other problem in our project look trivial.
On the Android platform, there's little in way of built-in solutions... Especially solutions that would let us stream and be OpenSource so we could integrate it into our project. Early on, we found IP Camera For Android, a project that promised h.264 streaming out of the camera. However it didn't work with our platform for reasons that were at that time quite nebulous.
So we temporarily gave up on "IP Camera for Android".
We then spent a larger chunk of time researching how to get ffmpeg (C++ OpenSource suite of video encoders used in many, many embedded products) to work on Android which required getting familiar with cross-compilation to the ARM platform. The result, sadly, even after exploring all avenues we could think of to access the low-level video4linux interface (/dev/video0) on the Haipad was just random noise.
So we then went back to IP Camera for Android and really understand how it worked so we could make the simple required fix...
Fundamentally, to understand the issue with the output generated by the Android encoder we have to understand that a video file is two things:
1) the encoded stream (using a codec, in this case H.264 AVC) and
2) the "container" of that content that contains information to describe the video stream.
In the case of the MP4 container format, that description comes into the form of "atoms" or "boxes" of information.
The problem is that the output produced by the Android encoder is not designed for streaming; this means that those "boxes" are not written until the stream has been fully written out to disk. This also means that the header information that would let a codec parse the output is not available until the end, which isn't practical when streaming video. In the case of H.264, the quite essential information we want is called the SPS (Sequence Parameter Set) and PSS (Picture Parameter Set (PPS)), arrays of bytes that serve as parameters to the H.264 AVC codec.
The nice hack that "IP Camera for Android" uses is to create a sample video on disk and read the SPS and PSS information from that prior to stremaing. Since the output between encoded videos at the same resolution coming from the same encoder is consistent, that same information can be reused for other video streams.
For streaming, "IP Camera for Android" captures the video streams by redirecting the encoder output to a local socket. That socket is read by "StreamingKernel" whose primary task is to skip the incomplete MP4 header and break down the stream into frames. The "Streamer" then inserts the previously read SPS and PSS and constructs a more streamable VLC output (VLC is another video container format). The VLC stream is made up of the individual H.264 frames wrapped in VLC "tags" served from an embedded WebServer to a flash based video player on the client side.
Yes, this means modern Android phone can run a webserver that serves real-time encoded video!
When you think about that, it's kind of pretty awesome to realize that we live in the future...
The original problem was that the software made some simple but erroneous assumptions about the format of both the sample video and the length of the MP4 header. Actual encoded video data starts after an "atom/box" called "mdat". The mdat actual position varies depending on what the encoder is producing - the original code set the header size at 32 bytes while post-2.1 Android devices encoders header length is 44 bytes...
As fixing the size of that header is annoying, we modified the code to read byte per byte from the stream until we get to mdat and then actually doing the streaming. In addition, we've made changes to reduce the number of costly data copies and temporary buffers to "streamline the streaming".
The result is still not entirely satisfactory due to the latency that seems to be introduced by buffering inside the encoder... But it's a big step forward.
This is quite a detour into the depths of Android and streaming. It did give us valuable experience with the platform and we can now use that experience as a building block for other interesting projects. But it also came at the cost of lots of time and energy that could have been used making progress elsewhere in the competition project...
So now we have to pick all the pieces we've done in one coherent whole...
You can follow or find more details on Github and the wiki as we try to put all the pieces together into a demoable final project. More updates on other aspects of the project soon...
Robot head mechanics assembled and moving!
Sideways and up/down movements of the tablet controlled by a simple servo sweep program in the Arduino Uno and powered by a rechargeable 6 volts batteries.
The servos are two Tower Pro MG995 metal gear servos, inexpensive at around 75 RMB each (USD$12).
Big thanks to these interns at the company I work at who spent two evenings figuring servo and brackets assembly. They had the forethought of buying a quite essential bag of longer M3 screws! Too bad they're going back to their home province today...:
(left to right)
- Danzel Lee (李劲)
- Swat Su (苏迪)
- Worlf Wang (王宇)
...also thanks to Mr Dremel (well, actually a Chinese cousin of Mr Dremel since it's a Chinese clone ^_^) who made this possible:
Yup, what you're seeing up there are four holes drilled in the back panel of the Haipad M701 (the IPad clone) so I could screw it securely to the servos bracket!
And yes, Android still boots after re-assembling in case you were wondering...
Oh, you want to see the innards of the Haipad M701 while we're at it? Here they are!
I'll upload videos and pictures with on Picasa and Youtube so check back for another blog post if you want to see those!
XinCheJian trying to find a suitable mechanical configuration for the head of the robot
Posted by rngadam Apr 12, 2011A real mechanical engineer would have sat down, calculated all the static and dynamic forces, selected appropriate servos for the task, designed in CAD, ordered custom parts and THEN assembled it. At XinCheJIan, we like to start with the end and work our way back ^_^. Seriously, if you're a mechanical engineer, we'd be happy to have your help - this is very far removed from our day jobs doing software! One issue we're having is figuring out how to attach the "neck" to the heads up/down motor. I think I've figured out by randomly mashing parts together that I'm supposed to attach one part to the servo horn (using the aluminium "adapter") and the other to the bearing on the other side. But I seem to be missing the right screw and nut to do that...
Gorilla Pod attached with tie wrap is a total hack but it (kinda) holding together while we figure out what custom part we'll use.
How the heck do I get both the bearing and the servo horn to be both attached to the bracket?
It's kinda fun to have random mechanical parts and try figure out without documentation how things fit with each others, but it's also taking longer than necessary and akin to apes randomly mashing rocks together to make tools...
Thanks to Min Lin's fantastic work at searching every tiny corner on the Chinese online shopping mall that is Taobao we got a whole bunch of stuff today!
Servos of various sizes, M3/M4 metric screws, brackets...
Total cost of what you see in the picture: about 600 RMB (USD$92)!
A fantastic deal that can be used in very many robots. But the sticker tag cost is only a small fraction of the real engineering cost since Min Lin spent days online to shop, discuss and negociate with the online vendors!
We'll now need to also do extensive experimentation to figure out what works and what doesn't.
- theknode | The Drawer
- You have to be logged in to view the content of the Drawer. Please Login or Register.