Skip to content
Spiderwiz

Spiderwiz

Let the wizard weave your web

Menu
  • About
  • Vision
  • Download
  • Tutorial
  • Javadoc
  • Blog
  • Contact

Lesson 12: Don’t Miss a Bit – Lossless Data Objects

Up till now we demonstrated the concept of live data object sharing. Data flows between applications that are up, running and connected. If any application goes offline, it issues Reset requests when it comes back and peer applications respond by re-sharing the objects that they have created. This works with indisposable objects, but disposable objects that are not consumed are, well, disposed.

There are, however, cases when disposable objects are not necessarily dispensable (who said food?). In these cases we do not want to lose data even when the consumer application is disconnected or inactive. There are excellent tools in the market for handling these scenarios, most notably Apache Kafka (and a Kafka plugin for Spiderwiz is on the way). Here we will describe a simpler solution that is built into the Spiderwiz framework. It is not as capable and trustable as Kafka – for instance it does not guarantee the order of the data feed – but it is sufficient in many cases, and in some cases even preferable because lost data objects are recovered in the background without interfering with the prompt delivery of live data.

In fact the Spiderwiz solution is so simple that it requires only one extra character. All you have to do in order to get lossless data objects is to append the ‘+’ character to the Object Code that you list in Main.getConsumedObjects(). We are going to show an example of it.

As a use case for this technology we picked the Chat Archiver of Lesson 8. Recall that this application listens to the entire chat activity, records it and replies to history requests by the chat applications. The way we did it, chat activity cannot be recorded when the Archiver is not up and running. We are going to amend the system so that ChatMessage objects will be considered “lossless”, which requires a backup mechanism to keep the objects when the Archiver is not active and recover them when it is up again.

As we have just mentioned, in theory all we have to do is to append the ‘+’ sign to ChatMessage.ObjectCode that is an element in the list that getConsumedObjects() returns. But before doing that there is a small architectural problem we need to tackle. The way we did it before, the Archiver joins every room that it is made aware of so that the routing mechanism of the chat applications includes it as a destination for every message. Obviously in order to join a room the Archiver needs to be active. Since now we want it to be a destination for all chat messages even when it is down, we need to find another solution.

Our solution is to use the appParams parameter of filterDestination() like we did in Lesson 4. The Archiver will override getAppParams() to map “role” to “admin”. The chat applications will route messages to applications with this mapping whether or not they are in the same room.

One more issue we have to remember is that in the previous lesson we added ChatApp between ChatRoom and Chatter and used the user name as the object ID of Chatter. The Archiver shall be amended accordingly.

So here is ArchiverMain:

package org.spiderwiz.tutorial.lesson12.archiver;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.spiderwiz.core.DataObject;
import org.spiderwiz.tutorial.objectLib.ChatApp;
import org.spiderwiz.tutorial.objectLib.ChatHistoryQuery;
import org.spiderwiz.tutorial.objectLib.ChatMessage;
import org.spiderwiz.tutorial.objectLib.ChatRoom;
import org.spiderwiz.tutorial.objectLib.Chatter;
import org.spiderwiz.tutorial.objectLib.ConsoleMain;

/**
 * Provides the entry point of the application. Initializes and executes the Spiderwiz framework.
 */
public class ArchiverMain extends ConsoleMain {
    private static final String ROOT_DIRECTORY = "";
    private static final String CONF_FILENAME = "archiver.conf";
    private static final String APP_NAME = "Chat Archiver";
    private static final String APP_VERSION = "Z1.01";  // Version Z1.01: Initial version
    static final String MY_USERNAME = "_chat_message_archiver";  // Use this as the "chatter" name when joining a room

    /**
     * Class constructor with constant parameters. Call super constructor and create the user map.
     */
    public ArchiverMain() {
        super(ROOT_DIRECTORY, CONF_FILENAME, APP_NAME, APP_VERSION);
    }
    
    /**
     * @return the class instance as ArchiverMain type.
     */
    public static ArchiverMain getInstance() {
        return (ArchiverMain)ConsoleMain.getInstance();
    }

    /**
     * Application entry point. Instantiate the class and initialize the instance.
     * 
     * @param args the command line arguments. Not used in this application.
     */
    public static void main(String[] args) {
        new ArchiverMain().init();
    }

    /**
     * @return an empty list
     */
    @Override
    protected String[] getProducedObjects() {
        return new String[]{};
    }

    /**
     * @return the list of consumed objects
     */
    @Override
    protected String[] getConsumedObjects() {
        return new String[]{
            ChatMessage.ObjectCode + ChatMessage.Lossless,  // Listens to every message for archiving.in lossless mode
            ChatHistoryQuery.ObjectCode                     // Replier of this query.
        };
    }

    /**
     * Add implementation classes to the object factory list of this application.
     * @param factoryList
     */
    @Override
    protected void populateObjectFactory(List<Class<? extends DataObject>> factoryList) {
        super.populateObjectFactory(factoryList);
        factoryList.add(ChatRoom.class);
        factoryList.add(ChatApp.class);
        factoryList.add(Chatter.class);
        factoryList.add(ChatMessageArchiver.class);
        factoryList.add(ChatHistoryQueryReply.class);
        factoryList.add(ArchivedUser.class);
        factoryList.add(ArchivedChatItem.class);
    }

    /**
     * @return a map with 'role'->'admin' mapping
     */
    @Override
    public Map<String, String> getAppParams() {
        return new HashMap<>() {
            {
                put("role", "admin");
            }
        };
    }
    
    /**
     * Process an input line - nothing to process in our case.
     * @param line      The input line
     * @return true
     */
    @Override
    protected boolean processConsoleLine(String line) {
        return true;
    }
}

Chatter is not produced any more (line 51) because we do not need to report that the application joins a room. For the same reason ChatRoom is not consumed (line 59). Also since we can now fetch the chatter name from the Chatter object ID, we can figure it out from the ChatMessage identification hierarchy and we do not need to consume Chatter any more.

The specification of ChatMessage.ObjectCode + ChatMessage.Lossless in line 61 means that ChatMessage is consumed lossless (the constant value of ChatMessage.Lossless is the plus symbol mentioned above).

We need to register the new ChatApp class in populateObjectFactory() (line 74).

And finally as we said, we override getAppParams() (line 86) to map “role” to “admin”.

There are two tiny modifications in ChatMessageArchiver:

package org.spiderwiz.tutorial.lesson12.archiver;

import org.spiderwiz.tutorial.objectLib.ChatMessage;
import org.spiderwiz.zutils.ZDate;

/**
 * Implementation class of ChatMessagee for the archiver.
 */
public class ChatMessageArchiver extends ChatMessage {

    /**
     * Every arriving message is encapsulated in ArchivedChatItem and archived.
     * @return true
     */
    @Override
    protected boolean onEvent() {
        try {
            ArchivedChatItem item =
                ArchiverMain.createTopLevelObject(ArchivedUser.class, getParent().getObjectID()).
                    createChild(ArchivedChatItem.class, null);
            item.archive(ZDate.now(), getParent().getParent().getParent().getObjectID(), getMessage());
        } catch (Exception ex) {
            ArchiverMain.getInstance().sendExceptionMail(ex, "When trying to archive a chat item", null, false);
        }
        return true;
    }
}

In line 19 we take the chatter name from the object’s parent ID rather than from its name property. In line 21 we take the room name from the object’s great-grandparent instead of its grandparent.

We do not need ChatMessageArchiver anymore so we delete it.

That’s all for Chat Archiver. There is no change in its configuration file.

The chat application from the previous lesson remains almost the same, except of ChatMessageImp:

package org.spiderwiz.tutorial.lesson12.chat;

import java.util.Map;
import java.util.UUID;
import org.spiderwiz.tutorial.objectLib.ChatApp;
import org.spiderwiz.tutorial.objectLib.ChatMessage;
import org.spiderwiz.tutorial.objectLib.ChatRoom;
import org.spiderwiz.tutorial.objectLib.Chatter;

/**
 * Consumer implementation of the ChatMessage class
 */
public class ChatMessageImp extends ChatMessage {
    /**
     * Called when a chat message is received. Check if the message has been sent in the room we are in,
     * then print the sender name and the message.
     * @return true to indicate that the event has been handled.
     */
    @Override
    protected boolean onEvent() {
        if (ChatMain.getInstance().isSameRoom(getParent().getParent().getParent().getObjectID())) {
            String name = ((Chatter)getParent()).getName();
            if (name == null)
                name = getParent().getObjectID();
            System.out.printf("%1$s: %2$sn", name, getMessage());
        }
        return true;
    }

    /**
     * Restrict message sending to:
     *      1. applications that are not the current application.
     *      2. applications that are on the same chat room as the current user, or:
     *      3. applications whose 'appParams' contains the mapping 'role'->'admin'
     * @param appUUID           destination application UUID
     * @param appName           destination application name
     * @param userID            destination user ID
     * @param remoteAddress     remote network address
     * @param appParams         application parameters of the destination application
     * @return true if the object should be sent to this destination
     */
    @Override
    protected boolean filterDestination(UUID appUUID, String appName, String userID, String remoteAddress,
        Map<String, String> appParams
    ) {
        return
            !ChatMain.getInstance().getAppUUID().equals(appUUID) && (
                isMemberOfMyRoom(appUUID) || appParams != null && "admin".equals(appParams.get("role"))
            );
    }

    /**
     * Check if the user running the application whose UUID is given is on the same room as this message is sent to.
     * @param appUUID   Application UUID to check
     * @return  true if the application is in the same room as us.
     */
    private boolean isMemberOfMyRoom(UUID appUUID) {
        try {
            // The room is the great-grandparent of the message
            ChatRoom chatRoom = (ChatRoom)getParent().getParent().getParent();
            return chatRoom.getChild(ChatApp.class, appUUID.toString()) != null;
        } catch (NoSuchFieldException | IllegalAccessException ex) {
            // Arriving here if a ChatApp does not contain ObjectCode static field or the field is not public.
            // In this case, send a command exception message.
            ChatMain.getInstance().sendExceptionMail(ex, "Cannot instantiate ChatApp", null, false);
            return false;
        }
    }
    
}

The change is in filterDestination() (line 43). As mentioned, we check if appParams parameter is not null and contains the “role” -> “admin” mapping. If it does, the message is delivered to the application even if its user is not in the same room.

Before running the test we need to modify the configuration files of the chat applications, as follows:

[application name]Chat 1
[log folder]/tests/Chat1/Logs
[backup folder]/tests/Chat1/Backup
[history file]/tests/Chat1/history.dat
[producer-1]websocket=localhost:90/MyHub
[application name]Chat 2
[log folder]/tests/Chat2/Logs
[backup folder]/tests/Chat2/Backup
[history file]/tests/Chat2/history.dat
[producer-1]websocket=localhost:90/MyHub
[application name]Chat 3
[log folder]/tests/Chat3/Logs
[backup folder]/tests/Chat3/Backup
[history file]/tests/Chat3/history.dat
[producer-1]websocket=localhost:90/MyHub

Each of the files contains two new properties – backup folder and history file. The first specifies the folder where lossless objects that are destined to offline applications are saved. The second specifies the name of the file where information about peer applications is saved so that it is available when any of them becomes offline. The defaults of these properties, if they are not explicitly defined, are backup and history.dat respectively under the application root folder. In our case we run multiple instances of the chat application and we want each instance to have its own backup and history location, so we define them accordingly.

To test the implementation do the following:

  1. Run the Chat Archiver.
  2. Run the chat applications.
  3. Type a few messages in each chat application.
  4. Terminate (by typing exit) the Archiver.
  5. Type more chat messages.
  6. Terminate one or more chat applications.
  7. Restart it.
  8. Type more messages.
  9. Restart the Archiver.
  10. Type more chat messages.
  11. Type !history in one or more chat applications.

You will see that full message history is displayed including messages that were typed when the Archiver was offline. They may not be in the correct order, though, but they will be there and that is what we wanted to demonstrate.

With this lesson we have fully covered the Spiderwiz Programming Model. In the next chapter we will take a look at some general features that help programmers and software maintenance staff get the most out of it – monitor execution, discover bugs, get notifications and solve problems.

Post navigation

Lesson 11: Import/Export – Interfacing with Legacy Frameworks
Lesson 13: Everything that Makes Maintenance Life Easier

Table of Contents

  • Lesson 1: Hello World
  • Lesson 2: Communication – Producers and Consumers
  • Lesson 3: Going World Wide with a WebSocket Hub
  • Lesson 4: Fine Grained Object Routing
  • Lesson 5: Persistent Data Objects and the Data Object Tree
  • Lesson 6: Walking Through the Data Object Tree
  • Lesson 7: Query Objects
  • Lesson 8: Streaming Queries and Data Object Archiving
  • Lesson 9: Manual Data Object Reset
  • Lesson 10: Asynchronous Event Handling
  • Lesson 11: Import/Export – Interfacing with Legacy Frameworks
  • Lesson 12: Don’t Miss a Bit – Lossless Data Objects
  • Lesson 13: Everything that Makes Maintenance Life Easier
  • Lesson 14: Monitor, Manage, Maintain and Control Your Applications with SpiderAdmin
  • Lesson 15: Writing a Communication Plugin
  • Lesson 16: Writing a Mail Plugin
  • Lesson 17: Writing an Import Handler Plugin
  • Lesson 18: Customizing SpiderAdmin
  • Where do We Go from Here?

Spiderwiz 2025 . Powered by WordPress