12 Replies Latest reply on May 15, 2019 4:56 PM by philertman

    Uses for a Raspberry Pi supercomputer


      Hey, this is my first post to Element14!

      I came here from The Ben Heck Show, so naturally, like other semi-geeks, I decided to embark on a bunch of Raspberry Pi projects.


      One of these projects is actually inspired from various other sources, and that's to build a small Beowulf Cluster out of Raspberry Pis. I'll start with a 16 node and upgrade to 32 afterwards.

      The reasoning? Nothing more than to tinker and cross "Make a supercomputer" off of the bucket list. I am happy to spend that money and literally have it sat at home looking good until I can finally get a use out of it.


      My question: As a general computer user and keen video editor, are there any uses I could get from the supercomputer? What could someone like me actually use this for? How would I go about doing so?

        • Re: Uses for a Raspberry Pi supercomputer

          AS a bit of fun then it's OK you are learning something good on you for actually doing something!


          How ever as a practical cluster it's totally useless as 32-64 of them is about as powerful as a normal PC !

          Also since there are currently no video editors for the Pi and also that you would need to have one that was cluster aware there are no applications available for your  Cluster as is.


          Either accept it as a learning toy or maybe get some PC hardware  use Linux +KVM then create some VMs then Cluster  them to get the same learning experience with out the difficulties of using the RPI and having a network switch  32++ PSUs etc etc


          Obviously creating VMs then clustering them will not improve performance but will give you the learning experience!



            • Re: Uses for a Raspberry Pi supercomputer

              I wasn't expecting there to be any video editing capabilities, but it's good to have confirmation


              I like Dale's idea of making a smaller one, now that I know that there really isn't much of a practical use for is. A simple little thing that could sit in my office would be cool.


              Beyond that of video editing, etc, what are the possibilities of hosting a Minecraft server for my ICT class? I'm an ICT teacher at a primary school near me, so it'd be cool to have something along them lines for them to be able to interact when we do redstone projects, etc. I understand that it's entirely possible with just the 1 Pi, but would I hit two birds with one stone if I built the "not-so-supercomputer" and hosted a server off of it?

              EDIT: Preferably with a GUI. I'm not very tuned when it comes to Linux, yet. I feel like I'm asking the impossible here, for some reason haha


              My other idea would just use the damned thing as an NAS

                • Re: Uses for a Raspberry Pi supercomputer

                  Obviously you could use these as a Minecraft server or in fact a server for almost anything else. However the whole point of a "Beowulf Cluster" or similar HPC clusters is not to run individual tasks such as servers but a single distributed task communicating over a standard messaging protocol for example MPI or PVM. Otherwise it's not a supercomputer but a bunch of normal computers stuck on a network.


                  For a single threaded app you may find that a mini ITX mobo is much more powerful, however if learning about message passing is the issue then you need to run that set of tasks not conventional server instances that do not use that method of comunications

                    • Re: Uses for a Raspberry Pi supercomputer

                      Quite a few words and phrases being thrown around that I've not yet been introduced to haha


                      I speak from the perspective of a person who has never approached anything of this caliber before. Heck, I've never used Linux or anything before buying my Pi about 2 years ago! So you will have to bear with me on alot of this.


                      So I understand that a Minecraft server would be a poor use for such a construction, but am I right in saying I could configure them to run together in a manner which allows a server to be ran, and if (In the odd circumstance) a Pi should fail or disconnect, the others could keep it running?


                      Remember, alot of this is for learning, not so much getting the best results. I'm going to build this cluster, regardless of what other options I have, but I also want to learn what uses it has in the average person's life For instance, what sort of tasks in an average computer users life could be distributed amongst this system?

                        • Re: Uses for a Raspberry Pi supercomputer
                          John Beetem

                          Andy Moor wrote:


                          ... I also want to learn what uses it has in the average person's life For instance, what sort of tasks in an average computer users life could be distributed amongst this system?

                          The short answer is that average computer users don't need a supercomputer.  This is why there are billions of PCs and tablets and smart phones, but few supercomputers.  The longer answer is that most of those billions of computers do have very powerful parallel GPUs which have the power of supercomputers from decades ago.


                          Supercomputers are best suited to "embarrassingly parallel" applications than can easily be split up into small tasks that can efficiently run in parallel.  Graphics is well-suited, as is weather prediction, finite-element modeling to replace wind tunnels, and quantum chromodynamics.  Most users don't do any of this except graphics.

                          • Re: Uses for a Raspberry Pi supercomputer

                            So in the case of minecraft servers they would all be running there own environments so they wouldn't easily operate in a manner where they  offered one common environment.


                            Webservers would be trivial you could do this either via an Apache Module or a load balancer


                            Clustering for resilience is not HPC (a Beowulf or any other type of such cluster) so an active passive arrangement for a database is not an HPC cluster


                            HPC is clustering for performance and has specific protocols such as MPI/PVM to do it..


                            Problem here is that for either type of cluster your software needs to be written to be aware of the clustering solution.


                            Prepare for brain pain   the above issues and many more consume the working life of many !!!


                            Try something like Blender which I believe as an HPC configuration to demonstrate rendering of frames across many nodes.


                            Again good luck!

                      • Re: Uses for a Raspberry Pi supercomputer

                        Now that it is 2018 and we now have the Raspberry Pi 3b+ the power of these little things have jumped up quite nicely, I have a cluster of 12 Pi3b+ with two added ClusterHat's adding 8 Pi0w's on top just for fun.


                        My thing is im just trying to find something fun I can task these things to do, I was looking at installing docker and seeing if I can get an interesting container swarmed over them.... Found it interesting that there is a Ubuntu container that can run swarmed but so far I've not seen anyone attempt that at all... 


                        Any suggestions?

                      • Re: Uses for a Raspberry Pi supercomputer

                        I have made a Pi cluster, it was a great learning experament but kind of useless. I finally took it apart and used the Pi's as other projects. The cost compared to what you get for an end result isn't worth the money, the experience is. I created a 5 node cluster which cost around $300, for a 16 node it will be around $1000. For just a bucket list check I would do a smaller one as once the master Pi is programmed any slaves can be added by cloning the SD card of the first slave. So 1 slave or 100, it isn't much harder. If you want something that looks real cool and will keep for show, it would be worth it.



                        • Re: Uses for a Raspberry Pi supercomputer
                          John Beetem

                          If you want to use RasPi as a supercomputer, you won't get very far using those ARM11 cores.  RasPi's real computing power is its VideoCore 4 GPU.  The ARM11 is just a weak I/O processor for the GPU.


                          If you want to learn a lot and want to see how much of that 24 GFLOPS you can actually use, take a look at the VideoCore 4 document starting here: http://blog.broadcom.com/chip-design/android-for-all-broadcom-gives-developers-keys-to-the-videocore-kingdom/

                          • Re: Uses for a Raspberry Pi supercomputer

                            I did build a 70 node cluster and I must say that I had to face a lot of errors while building the raspberry pi supercomputer. However, we did manage to build it successfully and we used it to solve an NLP problem. The problem that we solved was multi-document summarization. We summarised around 300 blog posts in parallel by making use of the message passing interface.


                            Now, obviously Raspberry Pis aren't that powerful but the purpose of the application was to demonstrate the concept of supercomputers in an educataional institution.

                            1 of 1 people found this helpful
                            • Re: Uses for a Raspberry Pi supercomputer

                              I was just looking to build a cluster. I wanted to process files from SETI using the Raspberry PI. They have versions of BOINC for all the different computing projects. It is a way for you to use your cluster and help with these huge math projects.