OPI One Cluster Processing

jadr2ddude

I am planning a project to experiment with the use of Orange Pi’s for cluster processing. This will be my sophomore research project for High Technology High School. My goal is to provide speed to an Intel server at much lower cost by using many Orange Pi Ones. When my project is complete, I will release a system for easily setting up, deploying, managing, and utilizing an Orange Pi cluster. Here are some possible tasks it could be used to speed up:

• Compilation (distcc)
• Web Serving
• Dynamic Web Resource Generation
• Distributed Neural Networks
• Simulations
• Storage Systems
  

This project will use a buildroot system to provide containerization and will also manage the networking. I will provide the compiled image, config, and patches for the buildroot system.

Details

I plan to use the USB OTG ports on Orange Pi One SBCs for networking. They will all connect to a USB 3.0 hub which will then connect to a router running OpenWRT. Because they are USB 2.0, each Orange Pi One will be able to transfer data at up to 450 megabits per second – 4.5 times faster than the Ethernet ports. The hub will also have a SSD connected via a SATA-to-USB adapter. Because USB 3.0 can carry 5Gbit/second, all of the SBCs can communicate at top speed simultaneously, but not with the SSD running at top speed at the same time. This is still plenty of routing power for most purposes.

Scalability

There will be another set of six OPIs plugged into the first set (total of twelve boards in my cluster). This can be repeated to add more computing power, but will have declining returns due to increasing latency and decreasing bandwidth per node. To further expand the cluster, the OpenWRT router can run a software defined network.

jadr2ddude

I will be using the g_ether driver.

jadr2ddude

trpaslikcz replied at 2016-5-2 02:37
Interesting. This way each SBC will appear as an USB network dongle to a connected host. Did you a ...

The SSD will be mounted on the OpenWRT router and the filesystem will be shared to the nodes. I will be testing the performance and CPU usage of the networking once I have the images compiled.

jadr2ddude

Then I will write a new driver/protocol.

bhgv

interesting theme.

a easiest way to build cluster/grid/botnet/skynet:

download and build inferno-os. it is a framework/system designed especially to develop distributed networks ( https://en.wikipedia.org/wiki/Inferno_(operating_system) ). it can run as framework on win/lin/mac OSes, x86/arm/sparc/raspberypi like and many others cpus. and it can run as OS itself. yes, it's a oldschool skynet software.

here the main free sources repo - https://bitbucket.org/inferno-os/inferno-os/

it's easy to buid, but here is a prebuilt package (inferno-owen.xz) - https://mega.nz/#F!ZshHFJ5J!8-TKqn9gTKZh6QybGKiAjg
unpack somewhere, go to inferno-os folder, run emu.sh

cluster/grid/botnet/shwarz.. hm.. others (easyest kind):

1) one (only one) of nodes (computers in the grid) should contain the registry of currently connected nodes. its IP address should be written to

*inferno-os/lib/ndb/local*

line:
```
infernosite=
...
...
   registry=<IP address of registry node>
```
on every node in your botnet.

2) you need start emu on the registry node, run shell (if it started in the graphical mode), and type in the shell:
```
ndb/cs #start the connection server
mount -A -c {ndb/registry} /mnt/registry #mount registry server to /mnt/registry (-A - without security)
listen -A -v 'tcp!*!registry' {export /mnt/registry&} #listen to network and export mounted registry service to just connected node
```

3) start emus on worker nodes and type in their shells:
```
ndb/cs
mount -A 'net!$registry!registry' /mnt/registry  #connect to registry node and mount exported registry
#the should register itself in the registry
myip=`{os hostname -I} #get self IP (command "os" runs the command/program of hosting OS, "hostname -I" is here)
grid/reglisten -A -r svc rstyx 'tcp!' ^ $myip ^ '!5678' {auxi/rstyxd&} #register node, run styx (network protocol) server, listen network for rcmd and export rstyxd. very complex command
```
initialisation has finished (all without security - -A key). howto use:

you can see all connected nodes from any connected node:
```
ls /mnt/registry
```

you can send a job to a node:
```
rcmd -A tcp!<IP of a worker node>!5678 <command to execute remotelly. it may contain os cmd>
```

you can send a job to all connected nodes:
```
for i in `{ndb/regquery -n svc rstyx} {
    rcmd -A $i <your command>
}
```

in the inferno-owen.xz has file gridlib with this examples written. to use it:
in the shell on registry node:
```
run gridlib
masterinit
```

on worker nodes:
```
run gridlib
workerinit
newcpu
```

now you can use rcmd (or rsplit to run a cmd on all connected nodes)

this was a simple, tutorial example

bhgv

..continue

how to exchange of data?

first, in inferno the file tree is a name tree as it contain all what you want and can be changed for each thread. as example 2 threads of single program can use completely different name trees where same paths will mean different files, devices, channels etc

when **rcmd** is called it sends the name (file) tree of thread-caller to remote thread (this communication, connection and name trees are local to pair of connected threads. other pc/threads doesn't see them). remote thread can access to name tree of thread-caller by path */n/client*
```
rcmd -A tcp!<IP address of remote pc already connected to this grid>!5678 ls /n/client  #you can see the file/name tree of thread-caller but sent by remote thread
```

how to create a named pipe mapped to name tree to communicate with remote
```
mkdir /tmp/my_pipe  #this folder will be the name of pipe
bind "#|" /tmp/my_pipe  # '#|' - is a internal pipe driver (all internal driver names of inferno starts from #)
                        # you can see them using *cat /dev/drivers
```
a named pipe with name */tmp/my_pype* was created. you can see it
```
ls /tmp/my_pipe  # you will see files/names *data* and *data1*
                 # data is the input of this pipe
                 # data1 is the output of this pipe
```

let's try to use just created pipe
```
rcmd -A tcp!<IP of remote>!5678 cat /n/client/tmp/my_pipe/data1 & #read the output of the pipe on remote pc
echo hello remote! > /tmp/my_pipe/data  # send a message via our pipe to remote pc
```
that's all for today.

jadr2ddude

That sounds good! I may try Inferno afterwards. My main plan is to use docker, because most of the software I will be testing with will work easily with just Linux (e.g. LAMMPS).

bhgv

@jadr2ddude
you can use linux, but inferno (as plan9, as qnx, they all used same ideas) was designed especially for building systems distributed by network.
where many computers from different places work as single computer.
inferno and plan9 designed by bell labs, the father of unix, C, C++ and other
these were military projects (yes, it was not a joke when i named it skynet. it was the real development and inferno planned to be a core of real network of military things (rockets, tanks, avions, your army knife))
because of military order they were closed long time.

inferno was designed for highest portability (for example it can run on top of other OS as framework or without OS on bare hardware)
and easy programming. it has 2 languages interpreted sh (my examples here in sh) and compiled limbo (it looks like something between C++, pascal/oberon with some very highlevel constructions like from python, and it has easy to use multithreading system with mutexes, channels etc embedded directly into the language (as go or xc))
it also includes graphical sourcelevel debugger like ddd (also was writen in limbo and can work without recompilation)

it can be compiled under linux (and windows and mac and other) in minutes.
it has as minimum 3 pluses
1. you can develop your program on any system you want and run on other system without recompilation as it is a virtual machine
2. it is significally faster then other VMs including java and .net because of internal architecture, and because of jit compilers for most of architectures (x86, arm, mips, sparc, power)
3. it takes less of resources, thanks to very good garbage collector (when i tested it on win ce it took only 7-8mb with gui environment and pair of programs runned)

also it provides many abilities now absent in regular OSes or programming languages (maybe excluding go - a son of limbo, and xmos xc - also looks like limbo)


a example - how to browse/parse webpages directly from shell:
```
ndb/cs   #start connection service

fn rd_foo { #foo to read a page
  id=$1

  {
    echo 'GET /orangepibbsen/forum.php?mod=viewthread&tid=1459&extra=page%3D1 HTTP/1.1' &&
    echo 'connection: close' &&
    echo 'host: www.orangepi.org' &&
    echo ''
  } > /net/tcp/$id/data  #send a regular GET query to server

  cat /net/tcp/$id/data  #output received from server page to terminal

#  var=`{read < /net/tcp/$id/data} #or catch it to variable for parsing as example
}

fn get_opi_forum {  #foo to connect to a server
  (null ip)=`{ndb/csquery net!www.orangepi.org!http}  #get the IP of the server
#  echo $ip  # you can see received IP

  {
    id=`{read}  #get the id of just opened tcp channel
    echo connect $ip > /net/tcp/$id/ctl  #connect tcp channel $id to your server

    rd_foo $id #run the foo to read a page
  } <> /net/tcp/clone #by opening this file new tcp channel creates, by closing closes
                      # <> - mean - open for read and write and no close till the end
                      # of all commands between {}
}

get_opi_forum  #do the main foo
```
(all examples till now were in sh. inferno allows to make much operations with sh then linuxes' bash, for example regular replace, building of GUIs)
(limbo runs faster then sh but it will be in the future)

you can call commands of hosting OS from sh using command "os" and send/receive data from it to inferno
```
os <command string of hosting os>
```

next time i try to write how write a extension to inferno callable as limbo module in C (inferno contain a special tool to help in writing of C extensions)

bhgv

PS i forgot. here is a ready to use inferno based distributed system
https://bitbucket.org/inferno-os/owen

some slides about inferno-owen:
http://www.resc.rdg.ac.uk/presentations/Inferno_25-03-04/
www.vitanuova.com/papers/ugrid.pdf

it already included in the package from my mega.nz. if you interested, i try to write a pair of examples how to use it

jadr2ddude

Yes I will consider also trying out inferno, but most of the stuff I need to run/test is in C/C++.