How to Configure a Spiderwiz Application

Every Spiderwiz-based application needs a configuration file in order to run and perform its tasks. The file must reside in the application's root folder and its name shall be provided in its Main class constructor.

An application configuration file is a text file that contains mapping of property names (the keys) to property values. Each line in the file has the format:

[property name]property value

For instance, a configuration file may look something like this:

    [application uuid]ff8197de-3f44-4d2e-b224-e49fc49b7e6a
    [application name]My First Spiderwiz Application
    [log folder]Logs
    [backup folder]Backup
    [history file]history.dat
    [producer-1]ip=localhost;port=22220
    [producer-3]websocket=localhost:8080/TestWebsockets
    [producer server-1]port=23333
    [hub mode]yes
    [mail system]smtp;server=smtp.gmail.com;user=zvilif@spiderwiz.org;pwd=dumptramp;port=465;ssl=true
    [from address]First Application Alerts<1stapp.alert@spiderwiz.org>
    [to email]Spiderwiz Administrator<spiderwiz.admin@gmail.com>
    [to exception email]Spiderwiz Programmer<spiderwiz.geek@gmail.com>
    [obsolescence time in hours]1

Some properties require a parameter list, i.e. a list of pairs key=value concatenated by a semicolon (;). (The assignment is optional - keywords that are not assigned to values are valid). In the above example:

    [mail system]smtp;server=smtp.gmail.com;user=zvilif@spiderwiz.org;pwd=dumptramp;port=465;ssl=true

Properties can be predefined or custom. Predefined properties are used by Spiderwiz framework for the configuration of its tasks. Custom properties are defined by the user and can be programmatically fetched by user code that looks like:

    String myCustomProperty = Main.getConfig().getProperty("my custom property");

Predefined and custom properties can be mixed in the same configuration file.

The following is a detailed description of all Spiderwiz predefined properties and their roles. There is also a detailed description of the parameters for every property that requires a parameter list.

Spiderwiz Configuration

Configuration Properties
Property	Value Description
`application uuid`	Specifies the application UUID. The value is automatically generated and appended to the configuration file the first time the application runs. Normally you should never touch this value by hand. However if you move the application to another disk location and you want to keep its full context (configuration, network connection identifiers, session history etc.) you can move the configuration file as is, including this property. Do not keep the same UUID in more than one application configuration, as this would mess the entire system up.
`application name`	Sets the application name. This value overrides the value given to `Main` class constructor. If the property is not defined, the programmed name is used.
`log folder`	Absolute or relative path of the root folder of the application log file system. A relative path is relative to the application's root folder. If the property is not defined the default `"logs"` path is used. See Spiderwiz Logging System.
`backup folder`	Absolute or relative path of the root folder of the backup system for lossless delivery. A relative path is relative to the application's root folder. If lossless delivery is used and the property is not defined the default `"backup"` path is used.
`archive folder`	Absolute or relative path of the root folder of the data object archive. A relative path is relative to the application's root folder. If archiving is used and the property is not defined the default `"archive"` path is used.
`history file`	Absolute or relative pathname of the file that saves network information (applications, connections etc.) to carry between sessions of this application. A relative path is relative to the application's root folder. The default pathname is `history.dat` under the root folder.
`producer-n`	Defines a network client connection when the application connects as a producer. Up to 99 connections may be configured, when n is a number from 1 to 99 (in any order, skipping permitted). For parameters and values used in this entry see Network Client Connection Configuration.
`consumer-n`	Defines a network client connection when the application connects as a consumer. Up to 99 connections may be configured, when n is a number from 1 to 99 (in any order, skipping permitted). For parameters and values used in this entry see Network Client Connection Configuration.
`producer server‑n`	Defines a network connection server when the application acts as a producer to incoming connections. Up to 99 producer servers may be configured, when n is a number from 1 to 99 (in any order, skipping permitted). For parameters and values used in this entry see Network Server Configuration.
`consumer server‑n`	Defines a network connection server when the application acts as a consumer to incoming connections. Up to 99 consumer servers may be configured, when n is a number from 1 to 99 (in any order, skipping permitted). For parameters and values used in this entry see Network Server Configuration.
`import-n`	Defines an import network connection. Up to 99 import connections may be configured, when n is a number from 1 to 99 (in any order, skipping permitted). For parameters and values used in this entry see Import Connection Configuration.
`loginput`	Specifies logging options for lines arriving over network connections. The value can be any of the following: `no` (or anything that starts with it) - no logging of input lines (the default). `raw` - input lines are logged as they are, i.e. with the objects compressed. `full` - input lines are expanded to show uncompressed objects. `verbose` - every line is logged twice - raw and full. Note that logging can be configured individually for specific connections, overriding the general property described here. See Client Configuration and Server Configuration below. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`logoutput`	Specifies logging options for lines output to network connections. The value can be any of the following: `no` (or anything that starts with it) - no logging of output lines (the default). `raw` - output lines are logged as they are, i.e. with the objects compressed. `full` - output lines are logged before compression to show uncompressed objects. `verbose` - every line is logged twice - raw and full. Note that logging can be configured individually for specific connections, overriding the general property described here. See Client Configuration and Server Configuration below. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`reconnection seconds`	Specifies how many seconds a connection client shall wait after a failed connection attempt before retrying. A negative value means reconnection shall not be attempted. The default value is 60 seconds.
`disconnection alert minutes`	Specifies how many minutes a communication handler (producer, consumer or import) shall wait for a reconnection after detecting a client disconnection before sending an alert mail. Used in conjunction with the `alert` parameter of the network client configuration. If the value is negative disconnection alerts are never sent. The default is 3 minutes.
`idle alert minutes`	Specifies how many minutes a communication handler (producer, consumer or import) shall wait for traffic after detecting an idle state before sending an alert mail. Used in conjunction with the `alert` parameter of the network client configuration. If the value is negative idle alerts are never sent. The default is 3 minutes.
`stream rate`	Specifies the number of elements per minute to send when data objects are streamed. Object streaming takes place 1. during Data Object Resetting, 2. during retransmission of Lossless Objects. The default value for this property is 30,000 items per minute.
`hub mode`	Set this property to any value that does not start with "no" (case insensitive) to indicate that the application runs as a *hub, i.e. it requests data objects* that it does not consume in order to route them down the stream to applications that do consume them.
`pass through`	Set this property to any value that does not start with "no" (case insensitive) to indicate that the application shall not keep in memory any data object it consumes even if it is not defined disposable.
`start of day`	Specifies the hour of the day that the application executes its daily task in the format `hh:mm`. The default value is `04:00`
`obsolescence hours`	Specifies the number of hours a disconnected application is considered alive before it becomes obsolete. This is relevant for 1. remembering a target application for delivery of Lossless Objects, 2. SpiderAdmin. The default value for this property is 24 hours.
`mail system`	Defines the mail system to use for sending notifications and alerts. For parameters and values used in this entry see Mail System Configuration. If the property is not defined mails will not be sent by the configured application. However the application can still delegate mail sending to another application that does have access to a mail system. See Main.ObjectCodes.EventReport.
`from address`	Specifies the sender address to use for sending notification and alert mails. The address format depends on the defined mail system.
`to email`	Specifies the address (or addresses) to use for sending notification and alert (excluding code exception alerts) mails to. The address format depends on the defined mail system.
`cc email`	Specifies the address (or addresses) to use for sending notification and alert (excluding code exception alerts) carbon copies to. The address format depends on the defined mail system.
`to exception email`	Specifies the address (or addresses) to use for sending code exception alerts to. The address format depends on the defined mail system.
`cc exception email`	Specifies the address (or addresses) to use for sending code exception alert carbon copies to. The address format depends on the defined mail system. If not defined then CCs will not be sent.
`exception alert rate`	Specifies the rate in minutes that non-critical code exception events should generate alert mails (see Main.sendExceptionMail()). A negative value makes every exception critical. The default value is 60 minutes.
`minimum disk space in gb`	Specifies the minimum free disk space, in gigabytes, that if the system running the application runs under, then the application sends a notification mail to the address specified by the `to email` property (or by delegation). Do not specify this property if such notification is not required.
`spideradmin`	Implies that this application shall connect to SpiderAdmin with the specified user token.
`deploy folder`	Specifies the folder to which SpiderAdmin copies deployment files for this application. The default is `Deploy` under the application's root folder.
`restart command`	Specifies the command that SpiderAdmin needs to use in order to restart the application, as a comma separated list of arguments to use when instantiating ProcessBuilder. The default value is `"cmd,/c,restart.bat"`, i.e. The Windows arguments for running "restart.bat" in the application's root folder.
`deploy command`	Specifies the command that SpiderAdmin needs to use in order to deploy the application, as a comma separated list of arguments to use when instantiating ProcessBuilder. To this list, SpiderAdmin will add the name of the deployment file as supplied by its user, which is supposed to be a parameter in the batch file that is activated by this command. The default value is `"cmd,/c,deploy.bat"`, i.e. The Windows arguments for running "deploy.bat" in the application's root folder.

Configuration of network client connections, defined by producer‑n or consumer‑n properties, requires a parameter list as defined in the following table. There are two types of parameters. Those of the first type define the transport mechanism. These include class, ip, port, infile, outfile and websocket. The other parameters assign general features to the defined connection and pertain to every transport.

With respect to the transport mechanism, this can be custom, implemented as a plugin, or predefined. The class parameter defines a custom plugin, which may use custom parameters in addition to the parameters described in the table. If the class parameter is not used then the other parameters mentioned above determine a predefined transport as described in the table.

As mentioned, a network connection needs to be configured as either a producer or a consumer. This is just for the sake of convenience and does not limit the data flow in any way. An application may send data objects over a connection defined as producer, and may receive data objects through a connection defined as consumer. However the identification of a connection between two applications as mainly producing or mainly consuming helps in having a clearer bird eye view of the network topology.

Network Client Connection Configuration
Parameter	Value Description
`class`	Fully qualified class name of a custom connection plugin.
`ip`	The IP address or host name of the target server. The use of both `ip` and `port` properties defines a TCP/IP socket connection.
`port`	Port number of the target server. The use of both `ip` and `port` properties defines a TCP/IP socket connection.
`infile`	Input file pathname of a file channel, when the connection is configured to read from and write to disk files. Usually when you use this property you would also use `outfile`. This is used mostly for debugging, when you would use the output of the `outfile` parameter, or logs of a previous application session, to extract input for offline debugging.
`outfile`	Output file pathname of a file channel, when the connection is configured to read from and write to disk files. Usually when you use this property you would also use `infile`.
`websocket`	Defines a WebSocket client. The value specifies the protocol and URI of the server to connect to in the formats `uri`, `ws:uri` or `wss:uri`. The URI is the server address followed by a slash (/) and the context path to connect to. The protocol can be `ws` or `wss`, or it can be omitted in which case `ws` is used. For example: `ws:spideradmin.com/admin`. Note that the server context path shall be the root path of the server application, as this is the path used by Spiderwiz WebSocket server implementation to form the server's WebSocket path.
`compress`	If used, specifies how output lines sent over this channel shall be compressed. The value can be any of the following: `no` (or anything that starts with it) - no compression (the default for disk file input/output). `zip` - physical compression using GZIP. `logical` - object-based compression. `full` - both physical and logical compression (the default for all communication types except disk file input/output). For a full explanation of the compression options see Spiderwiz Data Compression.
`alert`	If used, instructs the framework to send an alert message when the connection drops or becomes idle for a specific amount of time (as specified by configuration properties described above). The value of this parameter determines the entity responsible for detecting the event and alerting. The options are: `client` - the client side of a client-server connection. `server` - the server side of a client-server connection. `both` - both sides would send an alert message when they detect a failure. If the parameter is not used failure alerts will not be sent.
`loginput`	If used, overrides the general `loginput` property to specify channel-specific logging options for input lines. The value can be any of the following: `no` (or anything that starts with it) - no logging of input lines. `raw` - input lines are logged as they are, i.e. with the objects compressed. `full` - input lines are expanded to show uncompressed objects. `verbose` - every line is logged twice - raw and full. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`logoutput`	If used, overrides the general `logoutput` property to specify channel-specific logging options for output lines. The value can be any of the following: `no` (or anything that starts with it) - no logging of output lines. `raw` - output lines are logged as they are, i.e. with the objects compressed. `full` - output lines are logged before compression to show uncompressed objects. `verbose` - every line is logged twice - raw and full. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`flushlogs`	If specified (no assignment is needed), indicates that the file that logs the activity on this channel shall be flushed on each transaction. Very useful for debugging, but should be used with care since it slows performance down dramatically.
`user`	Attaches the specified user name to the connection. The value is conveyed to the connected server and is used by it in methods such as DataObject.filterDestination(). User IDs are also used by the framework to prevent data routing from applications that have an attached user ID to applications that have a different ID.
`bufferfile`	If used, specifies a file pathname for buffering output lines that cannot be sent over the communication line due to congestion. Without this parameter, lines that cannot be sent will be lost.

Configuration of network connection servers, defined by producer server‑n or consumer server‑n properties, requires a parameter list as defined in the following table. There are two types of parameters. Those of the first type define the server transport mechanism. These include class, port and websocket. The other parameters assign general features to the defined server and pertain to every transport.

With respect to the server transport mechanism, this can be custom, implemented as a plugin, or predefined. The class parameter defines a custom plugin, which may use custom parameters in addition to the parameters described in the table. If the class parameter is not used then the other parameters mentioned above determine a predefined server type as described in the table.

See above the comment about producers and consumers.

Network Server Configuration
Parameter	Value Description
`class`	Fully qualified class name of a custom server plugin.
`port`	The number of the port the server listens on. The use of this parameter without the `class` parameter defines a TCP/IP server socket.
`websocket`	Defines a WebSocket server. This parameter is a keyword that does not require a value assignment. At most two WebSocket servers can be defined for an application - one when the application acts as a producer to incoming connections (i.e. when this parameter is part of the value of a `producer server‑n` property) and one when it acts as a consumer (when this parameter is part of the value of a `consumer server‑n` property.
`compress`	If used, specifies how output lines sent over connections accepted by this server shall be compressed. The value can be any of the following: `no` (or anything that starts with it) - no compression. `zip` - physical compression using GZIP. `logical` - object-based compression. `full` - both physical and logical compression (the default). For a full explanation of the compression options see Spiderwiz Data Compression.
`loginput`	If used, overrides the general `loginput` property to specify server-specific logging options for input lines received from connections accepted by this server. The value can be any of the following: `no` (or anything that starts with it) - no logging of input lines. `raw` - input lines are logged as they are, i.e. with the objects compressed. `full` - input lines are expanded to show uncompressed objects. `verbose` - every line is logged twice - raw and full. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`logoutput`	If used, overrides the general `logoutput` property to specify server-specific logging options for output lines sent over connections accepted by this server. The value can be any of the following: `no` (or anything that starts with it) - no logging of output lines. `raw` - output lines are logged as they are, i.e. with the objects compressed. `full` - output lines are logged before compression to show uncompressed objects. `verbose` - every line is logged twice - raw and full. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`flushlogs`	If specified (no assignment is needed), indicates that the files that logs the activity of connections established on this server shall be flushed on each transaction. Very useful for debugging, but should be used with care since it slows performance down dramatically.
`gateway`	Normally an application connected to a Spiderwiz network would manifest its own remote address (used in methods such as DataObject.filterDestination()). However there are cases that this is unwanted, for instance if the manifested value is a local area network address. The use of the `gateway` keyword specifies that the remote address of an application connected through this server is the address that the server identifies and not the address that the application manifests.

Configuration of import connections, defined by import‑n, requires a parameter list as defined in the following table. There are two types of parameters. Those of the first type define the transport mechanism. These include class, ip, port, infile and outfile. The other parameters assign general features to the defined connection and pertain to every transport.

With respect to the transport mechanism, this can be custom, implemented as a plugin, or predefined. The class parameter defines a custom plugin, which may use custom parameters in addition to the parameters described in the table. If the class parameter is not used then the other parameters mentioned above determine a predefined transport as described in the table.

Import Connection Configuration
Parameter	Value Description
`class`	Fully qualified class name of a custom connection plugin.
`ip`	The IP address or host name of the target server. The use of both `ip` and `port` properties defines a TCP/IP socket connection.
`port`	Port number of the target server. The use of both `ip` and `port` properties defines a TCP/IP socket connection.
`infile`	Input file pathname of a file channel, when the connection is configured to read from and write to disk files. Usually when you use this property you would also use `outfile`. This is used mostly for debugging, when you would use logs of a previous application session to extract input for offline debugging.
`outfile`	Output file pathname of a file channel, when the connection is configured to read from and write to disk files. Usually when you use this property you would also use `infile`.
`name`	Identifies the connection by a unique name. The value assigned by this parameter can be retrieved by calling ImportHandler.getName().
`charset`	Specifies the character set to use when reading and writing text data from and to the channel. The default is UTF-8.
`loginput`	If the keyword exists, it specifies that all lines arriving over the connection shall be logged. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`logoutput`	If the keyword exists, it specifies that all lines output to the connection shall be logged. For a detailed description of the framework's logging system see Spiderwiz Logging System.
`alert`	If the keyword exists, it instructs the framework to send an alert message when the connection drops or becomes idle for a specific amount of time (as specified by configuration properties described above).

The configuration of the mail system defined by mail system property requires a parameter list. The class parameter defines a custom mail plugin, and this may use custom parameters different than those described in the following table. If the class parameter is not used then this property, if exists, defines an SMTP server and specifies its properties as described in the table.

Mail System Configuration
Parameter	Value Description
`class`	Fully qualified class name of a custom mail system plugin.
`server`	SMTP server address.
`port`	Server port to use.
`ssl`	The keyword, if exists, specifies the use of SSL protocol.
`tls`	The keyword, if exists, specifies the use of TLS protocol.
`user`	The username to use for logging into the mail server.
`pwd`	The password to use for logging into the mail server.