{"id":1001,"date":"2020-08-13T20:02:41","date_gmt":"2020-08-13T20:02:41","guid":{"rendered":"http:\/\/spiderwiz.org\/project\/?page_id=1001"},"modified":"2025-04-09T06:15:58","modified_gmt":"2025-04-09T06:15:58","slug":"lesson-12","status":"publish","type":"page","link":"https:\/\/spiderwiz.org\/project\/tutorial\/lesson-12\/","title":{"rendered":"Lesson 12: Don\u2019t Miss a Bit \u2013 Lossless Data Objects"},"content":{"rendered":"\n

Up till now we demonstrated the concept of live data object\nsharing. Data flows between applications that are up, running and connected. If\nany application goes offline, it issues Reset<\/em>\nrequests when it comes back and peer applications respond by re-sharing the\nobjects that they have created. This works with indisposable objects, but disposable<\/a> objects that are not consumed are,\nwell, disposed.<\/p>\n\n\n\n

There are, however, cases when disposable objects are not\nnecessarily dispensable (who said food?). In these cases we do not want to lose\ndata even when the consumer application is disconnected or inactive. There are\nexcellent tools in the market for handling these scenarios, most notably Apache Kafka<\/a>\n(and a Kafka plugin for Spiderwiz<\/a> is on the way). Here we will\ndescribe a simpler solution that is built into the Spiderwiz framework. It is\nnot as capable and trustable as Kafka \u2013 for instance it does not guarantee the\norder of the data feed \u2013 but it is sufficient in many cases, and in some cases\neven preferable because lost data objects are recovered in the background\nwithout interfering with the prompt delivery of live data.<\/p>\n\n\n\n

In fact the Spiderwiz solution is so simple that it requires\nonly one extra character. All you have to do in order to get lossless data\nobjects is to append the \u2018+\u2019 character to the Object Code<\/em> that you list in Main.getConsumedObjects()<\/a>. We are going to show\nan example of it.<\/p>\n\n\n\n

As a use case for this technology we picked the Chat Archiver<\/em> of Lesson 8<\/a>.\nRecall that this application listens to the entire chat activity, records it\nand replies to history requests by the chat applications. The way we did it,\nchat activity cannot be recorded when the Archiver is not up and running. We\nare going to amend the system so that ChatMessage<\/code>\nobjects will be considered \u201clossless\u201d, which requires a backup mechanism to\nkeep the objects when the Archiver is not active and recover them when it is up\nagain.<\/p>\n\n\n\n

As we have just mentioned, in theory all we have to do is to\nappend the \u2018+\u2019 sign to ChatMessage.ObjectCode<\/code>\nthat is an element in the list that getConsumedObjects()<\/code>\nreturns. But before doing that there is a small architectural problem we need\nto tackle. The way we did it before, the Archiver joins every room that it is\nmade aware of so that the routing mechanism of the chat applications includes\nit as a destination for every message. Obviously in order to join a room the\nArchiver needs to be active. Since now we want it to be a destination for all\nchat messages even when it is down, we need to find another solution.<\/p>\n\n\n\n

Our solution is to use the appParams<\/code> parameter of filterDestination()<\/a> like we did in Lesson 4<\/a>. The Archiver will override getAppParams()<\/a> to map \u201crole<\/code>\u201d to \u201cadmin<\/code>\u201d. The chat applications will route messages to applications with this mapping whether or not they are in the same room.<\/p>\n\n\n\n

One more issue we have to remember is that in the previous\nlesson we added ChatApp<\/code> between ChatRoom<\/code> and Chatter<\/code> and used the user name as the object ID of Chatter<\/code>. The Archiver shall be amended\naccordingly.<\/p>\n\n\n\n

So here is ArchiverMain<\/code>:<\/p>\n\n\n\n

package org.spiderwiz.tutorial.lesson12.archiver;\n\nimport java.util.HashMap;\nimport java.util.List;\nimport java.util.Map;\nimport org.spiderwiz.core.DataObject;\nimport org.spiderwiz.tutorial.objectLib.ChatApp;\nimport org.spiderwiz.tutorial.objectLib.ChatHistoryQuery;\nimport org.spiderwiz.tutorial.objectLib.ChatMessage;\nimport org.spiderwiz.tutorial.objectLib.ChatRoom;\nimport org.spiderwiz.tutorial.objectLib.Chatter;\nimport org.spiderwiz.tutorial.objectLib.ConsoleMain;\n\n\/**\n * Provides the entry point of the application. Initializes and executes the Spiderwiz framework.\n *\/\npublic class ArchiverMain extends ConsoleMain {\n    private static final String ROOT_DIRECTORY = "";\n    private static final String CONF_FILENAME = "archiver.conf";\n    private static final String APP_NAME = "Chat Archiver";\n    private static final String APP_VERSION = "Z1.01";  \/\/ Version Z1.01: Initial version\n    static final String MY_USERNAME = "_chat_message_archiver";  \/\/ Use this as the "chatter" name when joining a room\n\n    \/**\n     * Class constructor with constant parameters. Call super constructor and create the user map.\n     *\/\n    public ArchiverMain() {\n        super(ROOT_DIRECTORY, CONF_FILENAME, APP_NAME, APP_VERSION);\n    }\n    \n    \/**\n     * @return the class instance as ArchiverMain type.\n     *\/\n    public static ArchiverMain getInstance() {\n        return (ArchiverMain)ConsoleMain.getInstance();\n    }\n\n    \/**\n     * Application entry point. Instantiate the class and initialize the instance.\n     * \n     * @param args the command line arguments. Not used in this application.\n     *\/\n    public static void main(String[] args) {\n        new ArchiverMain().init();\n    }\n\n    \/**\n     * @return an empty list\n     *\/\n    @Override\n    protected String[] getProducedObjects() {\n        return new String[]{};\n    }\n\n    \/**\n     * @return the list of consumed objects\n     *\/\n    @Override\n    protected String[] getConsumedObjects() {\n        return new String[]{\n            ChatMessage.ObjectCode + ChatMessage.Lossless,  \/\/ Listens to every message for archiving.in lossless mode\n            ChatHistoryQuery.ObjectCode                     \/\/ Replier of this query.\n        };\n    }\n\n    \/**\n     * Add implementation classes to the object factory list of this application.\n     * @param factoryList\n     *\/\n    @Override\n    protected void populateObjectFactory(List<Class<? extends DataObject>> factoryList) {\n        super.populateObjectFactory(factoryList);\n        factoryList.add(ChatRoom.class);\n        factoryList.add(ChatApp.class);\n        factoryList.add(Chatter.class);\n        factoryList.add(ChatMessageArchiver.class);\n        factoryList.add(ChatHistoryQueryReply.class);\n        factoryList.add(ArchivedUser.class);\n        factoryList.add(ArchivedChatItem.class);\n    }\n\n    \/**\n     * @return a map with 'role'->'admin' mapping\n     *\/\n    @Override\n    public Map<String, String> getAppParams() {\n        return new HashMap<>() {\n            {\n                put("role", "admin");\n            }\n        };\n    }\n    \n    \/**\n     * Process an input line - nothing to process in our case.\n     * @param line      The input line\n     * @return true\n     *\/\n    @Override\n    protected boolean processConsoleLine(String line) {\n        return true;\n    }\n}<\/pre><\/div>\n\n\n\n
Chatter<\/code> is not\nproduced any more (line 51) because we do not need to report that the\napplication joins a room. For the same reason ChatRoom<\/code> is not consumed (line 59). Also since we can now fetch the\nchatter name from the Chatter<\/code> object\nID, we can figure it out from the ChatMessage<\/code>\nidentification hierarchy and we do not need to consume Chatter<\/code> any more.<\/p>\n\n\n\n
The specification of ChatMessage.ObjectCode\n+ ChatMessage.Lossless<\/em> in line 61 means that ChatMessage<\/code> is consumed lossless (the constant value of ChatMessage.Lossless<\/code> is the plus symbol\nmentioned above).<\/p>\n\n\n\n
We need to register the new ChatApp<\/code> class in populateObjectFactory()<\/a> (line 74).<\/p>\n\n\n\n
And finally as we said, we override getAppParams()<\/a> (line 86) to map \u201crole<\/code>\u201d to \u201cadmin<\/code>\u201d.<\/p>\n\n\n\n
There are two tiny modifications in ChatMessageArchiver<\/code>:<\/p>\n\n\n\n
package org.spiderwiz.tutorial.lesson12.archiver;\n\nimport org.spiderwiz.tutorial.objectLib.ChatMessage;\nimport org.spiderwiz.zutils.ZDate;\n\n\/**\n * Implementation class of ChatMessagee for the archiver.\n *\/\npublic class ChatMessageArchiver extends ChatMessage {\n\n    \/**\n     * Every arriving message is encapsulated in ArchivedChatItem and archived.\n     * @return true\n     *\/\n    @Override\n    protected boolean onEvent() {\n        try {\n            ArchivedChatItem item =\n                ArchiverMain.createTopLevelObject(ArchivedUser.class, getParent().getObjectID()).\n                    createChild(ArchivedChatItem.class, null);\n            item.archive(ZDate.now(), getParent().getParent().getParent().getObjectID(), getMessage());\n        } catch (Exception ex) {\n            ArchiverMain.getInstance().sendExceptionMail(ex, "When trying to archive a chat item", null, false);\n        }\n        return true;\n    }\n}<\/pre><\/div>\n\n\n\n
In line 19 we take the chatter name from the object\u2019s parent\nID rather than from its name<\/code> property.\nIn line 21 we take the room name from the object\u2019s great-grandparent instead of\nits grandparent.<\/p>\n\n\n\n
We do not need ChatMessageArchiver<\/code>\nanymore so we delete it.<\/p>\n\n\n\n
That\u2019s all for Chat\nArchiver<\/em>. There is no change in its configuration file.<\/p>\n\n\n\n
The chat application from the previous lesson remains almost\nthe same, except of ChatMessageImp<\/code>:<\/p>\n\n\n\n
package org.spiderwiz.tutorial.lesson12.chat;\n\nimport java.util.Map;\nimport java.util.UUID;\nimport org.spiderwiz.tutorial.objectLib.ChatApp;\nimport org.spiderwiz.tutorial.objectLib.ChatMessage;\nimport org.spiderwiz.tutorial.objectLib.ChatRoom;\nimport org.spiderwiz.tutorial.objectLib.Chatter;\n\n\/**\n * Consumer implementation of the ChatMessage class\n *\/\npublic class ChatMessageImp extends ChatMessage {\n    \/**\n     * Called when a chat message is received. Check if the message has been sent in the room we are in,\n     * then print the sender name and the message.\n     * @return true to indicate that the event has been handled.\n     *\/\n    @Override\n    protected boolean onEvent() {\n        if (ChatMain.getInstance().isSameRoom(getParent().getParent().getParent().getObjectID())) {\n            String name = ((Chatter)getParent()).getName();\n            if (name == null)\n                name = getParent().getObjectID();\n            System.out.printf("%1$s: %2$sn", name, getMessage());\n        }\n        return true;\n    }\n\n    \/**\n     * Restrict message sending to:\n     *      1. applications that are not the current application.\n     *      2. applications that are on the same chat room as the current user, or:\n     *      3. applications whose 'appParams' contains the mapping 'role'->'admin'\n     * @param appUUID           destination application UUID\n     * @param appName           destination application name\n     * @param userID            destination user ID\n     * @param remoteAddress     remote network address\n     * @param appParams         application parameters of the destination application\n     * @return true if the object should be sent to this destination\n     *\/\n    @Override\n    protected boolean filterDestination(UUID appUUID, String appName, String userID, String remoteAddress,\n        Map<String, String> appParams\n    ) {\n        return\n            !ChatMain.getInstance().getAppUUID().equals(appUUID) && (\n                isMemberOfMyRoom(appUUID) || appParams != null && "admin".equals(appParams.get("role"))\n            );\n    }\n\n    \/**\n     * Check if the user running the application whose UUID is given is on the same room as this message is sent to.\n     * @param appUUID   Application UUID to check\n     * @return  true if the application is in the same room as us.\n     *\/\n    private boolean isMemberOfMyRoom(UUID appUUID) {\n        try {\n            \/\/ The room is the great-grandparent of the message\n            ChatRoom chatRoom = (ChatRoom)getParent().getParent().getParent();\n            return chatRoom.getChild(ChatApp.class, appUUID.toString()) != null;\n        } catch (NoSuchFieldException | IllegalAccessException ex) {\n            \/\/ Arriving here if a ChatApp does not contain ObjectCode static field or the field is not public.\n            \/\/ In this case, send a command exception message.\n            ChatMain.getInstance().sendExceptionMail(ex, "Cannot instantiate ChatApp", null, false);\n            return false;\n        }\n    }\n    \n}<\/pre><\/div>\n\n\n\n
The change is in filterDestination()<\/a> (line 43). As mentioned, we check if appParams<\/code> parameter is not null<\/code> and contains the \u201crole<\/code>\u201d -> \u201cadmin<\/code>\u201d mapping. If it does, the message is delivered to the application even if its user is not in the same room.<\/p>\n\n\n\n
Before running the test we need to modify the configuration\nfiles of the chat applications, as follows:<\/p>\n\n\n\n
[application name]Chat 1\n[log folder]\/tests\/Chat1\/Logs\n[backup folder]\/tests\/Chat1\/Backup\n[history file]\/tests\/Chat1\/history.dat\n[producer-1]websocket=localhost:90\/MyHub<\/pre><\/div>\n\n\n\n
[application name]Chat 2\n[log folder]\/tests\/Chat2\/Logs\n[backup folder]\/tests\/Chat2\/Backup\n[history file]\/tests\/Chat2\/history.dat\n[producer-1]websocket=localhost:90\/MyHub<\/pre><\/div>\n\n\n\n
[application name]Chat 3\n[log folder]\/tests\/Chat3\/Logs\n[backup folder]\/tests\/Chat3\/Backup\n[history file]\/tests\/Chat3\/history.dat\n[producer-1]websocket=localhost:90\/MyHub<\/pre><\/div>\n\n\n\n
Each of the files contains two new properties – backup folder<\/code> and history file<\/code>. The first specifies the folder where lossless objects\nthat are destined to offline applications are saved. The second specifies the\nname of the file where information about peer applications is saved so that it\nis available when any of them becomes offline. The defaults of these\nproperties, if they are not explicitly defined, are backup<\/code> and history.dat<\/code>\nrespectively under the application root folder. In our case we run multiple\ninstances of the chat application and we want each instance to have its own backup\nand history location, so we define them accordingly.<\/p>\n\n\n\n
To test the implementation do the following:<\/p>\n\n\n\n
  1. Run the Chat Archiver<\/em>.<\/li>
  2. Run the chat applications.<\/li>
  3. Type a few messages in each chat application.<\/li>
  4. Terminate (by typing exit<\/code>)\n     the Archiver.<\/li>
  5. Type more chat messages.<\/li>
  6. Terminate one or more chat applications.<\/li>
  7. Restart it.<\/li>
  8. Type more messages.<\/li>
  9. Restart the Archiver.<\/li>
  10. Type more chat messages.<\/li>
  11. Type !history<\/code>\n     in one or more chat applications.<\/li><\/ol>\n\n\n\n
    You will see that full message history is displayed\nincluding messages that were typed when the Archiver was offline. They may not\nbe in the correct order, though, but they will be there and that is what we\nwanted to demonstrate.<\/p>\n\n\n\n
    With this lesson we have fully covered the Spiderwiz\nProgramming Model. In the next chapter<\/a> we will take a look at some general\nfeatures that help programmers and software maintenance staff get the most out\nof it \u2013 monitor execution, discover bugs, get notifications and solve problems.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Learn how to guarantee that every object arrives at its destination even if there is a communication break or the destination application is temporarily inactive.<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":145,"menu_order":12,"comment_status":"closed","ping_status":"closed","template":"tutorial-child.php","meta":{"footnotes":""},"class_list":["post-1001","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/pages\/1001","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/comments?post=1001"}],"version-history":[{"count":8,"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/pages\/1001\/revisions"}],"predecessor-version":[{"id":1425,"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/pages\/1001\/revisions\/1425"}],"up":[{"embeddable":true,"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/pages\/145"}],"wp:attachment":[{"href":"https:\/\/spiderwiz.org\/project\/wp-json\/wp\/v2\/media?parent=1001"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}