Always in Sync
A Deep Dive into Novell Data Synchronizer
Written by Ken Baker
As you read in the June issue of Novell Connection Magazine (See Higher Levels of Collaboration), Novell Data Synchronizer keeps applications and mobile devices constantly in sync so users can always have access to the data they need. That article talked about the product’s mobility pack, its many-to-many synchronize engine and how to configure connectors. This article takes a deeper technical dive into the way connectors facilitate synchronization, as well as a few other supporting architectural elements.
Written to application-specific or standard APIs, the connectors enable Data Synchronizer to understand how and what needs to be done with data to be synchronized.
Synchronization Channels and Filters
The majority of the intelligence in Novell Data Synchronizer resides in its connectors. Written to application-specific or standard APIs, the connectors enable Data Synchronizer to understand how and what needs to be done with data to be synchronized. For example, the SharePoint connector uses the SharePoint Web Services API to access data and push it through the system. The Salesforce.com connector uses the Salesforce.com Web services API. Similarly, the GroupWise connector utilizes the SOAP interface to synchronize e-mail, tasks, calendar data and contact information. The use of standard or native APIs allows each connector to be built specifically for its target application.
Each connector in Novell Data Synchronizer is comprised of two main components: channels and filters. Channels conduct the flow of data from the synchronization engine through the connectors to the connected systems. Filters act upon the data—translating and manipulating the data in a prescribed manner.
At a very high level, channels and filters work together in the following fashion:
- As data moves through the Novell Data Synchronizer channels between two or more applications, the Novell Data Synchronizer connectors act as code converters and data filters.
- The data flows through the Novell Data Synchronizer system in the form of XML files that are either in an application-specific format or an application-neutral format.
- As data moves from the application toward the synchronization engine, it will initially be in an application-specific XML file format.
- As the application-specific XML file travels through the channel, the connector filter will translate the XML file into an application-neutral format.
- After it’s translated into an application-neutral format, the XML file is stored in the synchronization engine database.
- The application-neutral XML files stored in the synchronization engine database can be consumed by different application-specific connectors (i.e., GroupWise, SugarCRM, Sharepoint, Salesforce.com), which in turn can translate them into the appropriate application-specific format as they transfer them to their associated application.
If you're using Novell identity and security products, you’ll likely recognize that this design is very similar to that of Novell Identity Manager. While Novell Identity Manager synchronizes identities from different sources and targets, Novell Data Synchronizer synchronizes data in a similar fashion. These similarities are perhaps most notable in the use of a two-channel system with a source channel for outgoing events and a SINK channel for incoming events. In Novell Data Synchronizer, these channels exist in both the synchronization engine and the individual connectors.
The source channel in the synchronization engine is responsible for taking item or event data from its cache and passing it to the connector. After the connector translates the data into the appropriate application-specific format, the connector’s SINK channel will pass that information to the application. For data coming from an application to the engine, the connector’s source channel has responsibility for taking data from the application, while the engine’s SINK channel will place the application-neutral formatted data into its cache.
In addition to determining what data types are synchronized, filters have responsibility for managing the synchronization of changes that occur to data items within a connected application, system or device.
How to Synchronize
In addition to having translation filters, connectors have synchronization filters. Depending on application needs and purpose, each application will have different types of data that it will allow to be synchronized. Some of the common data types (referred to as data items) include e-mail messages, appointments, tasks, notes, address books, contacts and folders. Connectors will not necessarily support all the same types of data items. In other words, data item synchronization will vary based on the functionality and needs of the application.
In addition to determining what data types are synchronized, filters have responsibility for managing the synchronization of changes that occur to data items within a connected application, system or device. These changes (known as events) might be represented by actions such as an item addition, modification, move or deletion.
As data items and events pass through a connector, various filters applied to the data determine what and how the data is synchronized. Depending on the connector design, you can often use filters to determine which data items and events you want to synchronize.
Not only can you use filters to screen for certain content or fields, but you can use them to transform one data type into another data type, such as turning task data into e-mail data. Filter behavior can also be modified based on certain triggers. For example, you can have a trigger that causes a filter to treat a data item differently based on where that data was originally stored on the source application.
You configure a connector’s filters through the Novell Data Synchronizer Web Admin console. For example, for the GroupWise connector you can choose from a list of add, modify, move and delete events that it will synchronize. (See Figure 1.)
More on Filters
One of the standard filters in Novell Data Synchronizer provides a circular sync check. As data passes from the connector to the synchronization engine, this filter makes sure that it’s not synchronizing the same data over and over again in a circular fashion. This circular check filter prevents the creation of duplicate information.
The default filters that come with a connector are typically sufficient to enable the translation of data from an application-specific format into an application-neutral format and vice versa, as well as apply other basic filter operations. However, since these filters are made up of XSLT files, if you’re familiar with XSLT you can manually configure them if necessary. (See Figure 2.)
While you can define additional filter functions using the connector's custom namespace, the following are standard functions provided in the Novell Data Synchronizer namespace:
- def logMessage(self, context, message, level='warning'): log a message
- def getSettingsForTarget(self, context, targetDn): returns a user's settings
- def getConnectorSettings(self, context): returns a connector's settings
- def getEngineSettings(self, context):returns the sync engine's settings
- def getConfigEngineSettings(self, context): returns config engine's settings
- def getMatchingUserDN(self, context, sourceName): returns the dn of a user with a unique cn provided
- def dnExists(self, context, dn): checks if the dn is a valid dn in the tree
- def isTargetEnabled(self, context, dn): checks if the user is in the user table for the connector
- def base64Encode(self, context, message):
- def base64Decode(self, context, encodedMessage, returnAsNodeset=False):
- def stripHTML(self, context, text, unescapeFirst=False): removes html from the text
- def itemIDToObjectID(self, itemID):
- def objectIDToItemID(self, objectID):
Other Novell Data Synchronizer Components
In addition to the synchronization engine, connectors, channels and filters, the following three components play a critical role in the operation of the Novell Data Synchronizer system:
- Web Administration Service provides a Web-based interface called Synchronizer Web Admin to facilitate administration and management of the Data Synchronizer system. It allows you to add and remove connectors, as well as add or remove users for specific connectors and configure their individual synchronization settings. Users can also use the Synchronizer Web Admin to configure and control synchronization settings for each of their connected applications and devices.
- Synchronizer Configuration Engine provides communication between the Synchronizer Web Admin and the synchronization engine. The configuration engine passes configuration information from the synchronization engine to the Synchronizer Web Admin for viewing. It also passes configuration changes back to the synchronization engine for implementation.
- Connector Manager provides communication between the synchronization engine and connectors.
Novell Data Synchronizer also includes LDAP integration. While not required, Novell Data Synchronizer can leverage an LDAP directory to further facilitate user and group management.
The synchronization engine, connectors, filters, channels, open development environment and other architectural components of Novell Data Synchronizer all combine to give you the real-time synchronization that your users need among their diverse applications and mobile devices.
In addition to the connectors that Novell provides, Novell gives you a number of tools to facilitate the development of additional connectors. The first of these tools is the software development kit for Novell Data Synchronizer, which guides you in how to take advantage of the product’s open API.
Novell also provides a generic SOAP connector, which accesses Novell Data Synchronizer using standard SOAP calls. Primarily intended to be used as an example connector, the SOAP connector provides a guide or reference point for someone who wants to develop a connector. You can use the connector as a model to see how the synchronization engine transforms data coming out of the GroupWise system.
As an additional connector development tool, Novell also provides a flat file connector. The flat file connector can take item and event data cached in the synchronization engine, translate it into an XML format administrators can read, and push it to a specified directory or folder on the network. Once placed in the specified folder, it can be consumed by other applications or manually accessed by users. (See Figure 3.)
The flat file connector also has the ability to periodically poll a specified folder for newly stored data. When it identifies new data, it can grab that data and pass it back to the synchronization engine for consumption by connected applications or devices.
While it operates on a fairly simple and straightforward concept, its inherent flexibility and open nature make the flat file connector quite powerful in helping you extend the capabilities of Novell Data Synchronizer in a variety of ways. In essence, it gives you a semi-technical programming interface that can help you customize and extend the capabilities of Novell Data Synchronizer.
Leveraging the connector, you could create a custom application that when a certain event happens, it generates information in an XML file format and then stores it in a specified folder. Once in the folder, the flat file connector can grab it, transform it and store it in the synchronization engine where it can be used by other connected applications.
Another possible use would be to use the flat file connector to retrieve and forward reports generated by a certain application. The application could generate the report as an XML file and store it in a specified folder. As the flat file connector polls that folder, it would discover the new report, transform it into an application-neutral format, and store it in the synchronization engine database where it can be consumed by other connected applications or sent to a specific user.
The flat file connector could also be used to pre-populate a new user’s GroupWise, SharePoint, or SugarCRM system with standard information. To accomplish this, you might manually create a file in the proper XML format with the desired information and then store it in a specified directory. The flat file connector can then grab, transform and store that data in the synchronization engine cache. When a new account is created for a user, its application-specific connector can then automatically pull that data from the synchronization engine and pre-populate the user’s account accordingly.
Syncing It All Together
The synchronization engine, connectors, filters, channels, open development environment and other architectural components of Novell Data Synchronizer all combine to give you the real-time synchronization that your users need among their diverse applications and mobile devices. They work together to help you improve your organization’s collaboration efforts through seamless and real-time synchronization of e-mail, calendar, contact, event and other collaboration data residing in your business and collaboration systems. Whether in the office, at home or on the road, Novell Data Synchronizer can help ensure that your users always have access to the right information in the right place at the right time.
To learn more visit www.novell.com/datasync.