Spiderwiz promise is to \u201cfree the\nmassive programming resources that are required just to move the data around\nrather than dealing with the data itself\u201d. In this post we will take a close\nlook at how this is done within the Spiderwiz runtime engine.<\/p>\n\n\n\n
Moving data from producers to\nconsumers in a distributed system includes four aspects:<\/p>\n\n\n\n
All of these challenges are\ntackled by the Spiderwiz runtime transparently, and in the view of a programmer\nthat uses the Spiderwiz API \u2013 effortlessly. Let\u2019s see how it happens.<\/p>\n\n\n\n
Serialization<\/em><\/strong>. As explained in the previous\npost<\/a>, the data in a Spiderwiz system\nconsists of Data Objects<\/em> that are\ndescribed in a Data Object Class Library<\/em>.\nA data object class contains property definitions and annotations<\/a> that\ndictate the serialization format of an object of the specified type. The class\ndefinition, used by both producers and consumers, forms the interface that\nbinds the endpoints to the same serialization format. Programmers need only commit<\/a> changes at the\nproducer side and handle events<\/a> at the consumer side and the runtime does everything in\nbetween.<\/p>\n\n\n\n Delivery<\/em><\/strong>. A Spiderwiz-based network is a network of Spiderwiz-based\napplications, which are connected via some sorts of communication channels that\nare supported by the runtime. Currently the supported channels are TCP\/IP\nsockets, WebSockets and sequential disk files (the latter mainly for\ndebugging). Other types of channels can be added as Communication\nPlugins<\/a>. Any network topology is allowed,\nprovided that consumers of specific data types are connected to the producers\nof that type either directly or through one or more applications that are\nconfigured as a \u201chub\u201d. Network connections are defined in the configuration files<\/a> of\nthe applications and do not require programming.<\/p>\n\n\n\n Data routing is also based on the\nData Object<\/em> paradigm. Every\napplication that starts up manifests the data\nobject types<\/em> that it produces and the types that it consumes. The runtime\nengines (in plural, because there is an engine at the core of each application\nand the engines cooperate) match producers and consumers and route the data\nbetween them. There is no need for more than one communication path between two\nendpoints. Data objects of different types are multiplexed into a single data\nstream, while the \u201chub\u201d applications demultiplex, reroute and remultiplex them\nas necessary.<\/p>\n\n\n\n As explained in the previous\npost, data delivery is not limited to one-to-one communication paths, therefore\nthe routing mechanism described here can use unicast\nor multicast as necessary.<\/p>\n\n\n\n Integrity<\/em><\/strong>. The complex data delivery mechanism described above needs\nto cope with a range of challenges, of which data integrity is a major one.\nSince data routing is done on a per object basis, and considering that network\ntopology is unrestricted and can easily turn into a mammoth spaghetti, there is\na need to ensure that a stream of data objects is reconstructed at the consumer\nside in the same order as it is produced, with no duplications, definitely\nwithout endless circulation throughout the network, and that if a deficiency is\ndetected then the producer of the missing objects is notified and completes them. This is exactly what Spiderwiz runtime engine does.<\/p>\n\n\n\n Efficiency<\/em><\/strong>. Although network capacity nowadays seems to be endless,\nthere are many situations in which bandwidth is still a concern. This is the\ncase when remote sites are connected through communication lines that are\nrelatively slow, as in the World Wide Web in general. We all know that, despite\nthe spectacular surge in communication speeds, digital media \u2013 DTV, YouTube,\nNetflix and all these fabulous services that fill up the cyber space \u2013 would\nnot be what they are without video compression that reduces frames of thousands\nof bits each to few bytes.<\/p>\n\n\n\n This is a domain in which\nSpiderwiz has a striking impact. The split of data streams to small units of Data Objects<\/em> allows wise distribution\ntechniques, such as the multiplex and multicast described above, that substantially\naffect the overall network bandwidth.<\/p>\n\n\n\n But this is just the beginning.\nThe outstanding efficiency of Spiderwiz as a bandwidth shrinker comes from the\nway it exploits the concept of object-based\ndata compression<\/em>. <\/p>\n\n\n\n Properties of objects in the real world \u2013 location of a\ncar, estimated arrival time of a flight, stock value in a stock exchange\nmarket, to name a few \u2013 tend to change gradually over time in small\ndiscrepancies, if at all. As Spiderwiz is about distribution of data objects<\/em> that represent real world\nobjects, it applies a mechanism that compresses data object streams by tracking\nchanges in individual objects and encoding them on the wire, much like video\nstreams are compressed by tracking and encoding the discrepancies between\nsuccessive frames.<\/p>\n\n\n\n Consider for instance a service that is used to retrieve\nthe estimated arrival time (ETA) of every commercial flight around the globe.\nTo do its work, the service needs to communicate with the system of every\nairline in order to obtain the information relevant to that airline. These, in\nturn, need to communicate with each of their airplanes to obtain the\ninformation for the specific flight. Using standard protocols, the way to do\nthat is to poll every airline at a frequency that depends on how much we want\nthe data to be up to date, and get back a dump of the data for the entire\nairline fleet. Assuming that there are about 50,000 flights worldwide every\nday, and that the data size for every flight, including overhead, is at least\n50 bytes, then we have at least 2.5M bytes on the wire on each poll. A polling\nfrequency of 10 seconds would make it 15M bytes running on the wire every\nminute, which is equal to 2.5M bits per second.<\/p>\n\n\n\n In practice most services do not work like that. Services\nlike the Air Traffic Organization (ATO) implement proprietary interfaces and\nprotocols that allow data streaming in a much more efficient way. But\nproprietary protocols need cooperation on both sides, which is not always\nfeasible, and they require a lot of development work on both sides on a per\nproject basis.<\/p>\n\n\n\n The problem is solved entirely if all the involved parties\nuse Spiderwiz. To take the ETA service example above, how many ETAs around the\nworld are changed in average within one minute? 100? 200? Definitely not more\nthan that. And since we talk about conveying discrepancies, each update would\nnot take more than a few bytes. All together, we will not have more than one\nthousand bytes per minute instead of the 15 millions calculated above. That\u2019s a\ncompression ratio of %0.0067, and the data is always up to date by a fraction\nof a second. All this without any need for cooperation, and without writing a\nsingle line of code!<\/p>\n\n\n\n Another thing that happens under the hood of the Spiderwiz\nruntime engine is thread management. As an event-driven engine, it allocates\nseparate execution threads to handle different kinds of events. One of the most\nresource consuming operations in Java is the spawning of new execution threads.\nFor this reason, the engine never spawns threads on the fly. All of them are\ncreated during application initialization, and a sophisticated queuing\nmechanism is applied in order to execute all tasks quickly, efficiently and\nwith the least resource consumption as possible. This is too transparent to the\nprogrammers, who would rarely need to spawn their own execution threads.<\/p>\n\n\n\nObject-Based Data Compression<\/h2>\n\n\n\n
Thread Management<\/h2>\n\n\n\n