HCE and Distributed Remote Command Execution

General definitions

The Distributed Remote Command Execution (DRCE) functionality provides hce-node application clients with the possibility to execute Linux shell commands, to spawn process via pipe, and to interact with the spawned process by std input, output, error as well as by file. Flexible protocol defines custom way to interact with the process, to give him incoming data and to collect produced data. By this service handler integrated inside the hce-node application any kind of computational task can be executed remotely on the low level replica or shard nodes of the cluster hierarchy. Most typical way of possible usage of DRCE:

• Execute data processing task, collect and fetch results from down replica or shard nodes according with cluster hierarchy.
• Manage some Linux host systems remotely by group actions that can be spread wide on correspondent cluster infrastructure.
• Spread some data to target cluster nodes for immediate or farther processing.
• Fetch some data from many target host systems, like logs and collect them at client side.
• Any kind of recombination of all above.

Main advantage of DRCE it is that structure of cluster for requests distribution, tasks execution planning, results collection and so on - can be custom defined and changed at run-time period.
Execution units can be connected at any time and extend power of processing pool. Also, execution units can be started on different physical servers and united in balance mode. It is very important to note that request message handling inside one unit is single-threaded, so no any mutexes or blocking as well as asynchronous request/response TCP sockets operation provided with ZMQ sockets library. On schema below simple DRCE computational cluster schema illustrate the connections direction and behavior of nodes in different roles:

Implementation of DRCE functionality in hce-node package includes client-side API for php language and client application to test and benchmark some algorithms and execution environment.
The Demo Test Suit includes set of prepared requests in json format that can be extended and changed manually. This set covers most important for hce-node execution environment like php and java, as well as provided with base Debian OS distributions like python, ruby and perl. Because very good affinity of hce-node application process instance and CPU kernel - the pool of hce-node replicas can to utilize CPU and I/O resources effective way provides up to 800 rps on one four kernel i5 physical server. On screenshot below the load level and by CPU kernel distribution shown.

As that can be seen above, for very short task that has execution time about 1 ms and 16 client threads - four computational units loads four CPU kernels in most effective way. Note, that in this test clients that requested tasks (drce.php application) and HCE cluster nodes instances located on the same physical host server. This is test #00 of Demo Tests Suit package of DRСE tests1.
The client application drce.php supports cli interface and provide usage of prepared jasons in many different tests and demos. Request and response protocols are very simple and formalized in json format. So, can be easy implemented on many other languages that are supported by ZMQ library.
The DRCE it is a second functionality after the Sphinx search that is natively supported by HCE.