How to use Ouiji

Project Files

To start your project, you will need to create a project file. Similar to most IDEs, the project file will help you organize separate multiple project concurrently and over development instances. Project files store project settings as well as 'Secondary Data' to be used in data mergers.

When you open Ouiji, you will see the options "Create Project" & "Open Project". Click the desired action.

Home Screen

Once you have an open project, the home screen will display six options. These may be considered steps for a new project, but is a quick navigation within existing and predefined projects.

Application Navigation & Menu

At the top left of the Ouiji Application you will find the application. This menu will allow you save, close and quit the application and well as allow you to change the visual style or 'theme' of the application. Directly below the 'File' menu is a small navigation button. Clicking this button will open a quick menu, allowing you to navigate to different application sections.

Setting Up Data Catalogs

The database catalog screen requires the following fields:

1. Name:  Requires a unique name which Ouiji will refer to when referencing the catalog entry.

2. Data Server: The data server on the network where the data catalog reside.

3. User Name / Password: An authorized user and password pair of the database catalog.

4. Connection String: The connection string used to log into the database catalog. Note that the bracketed { } terms will be replaced with the correlated fields. The default connection string should not be modified unless necessary.

5. Save Login: The save login checkbox will store an encrypted version of the username and password within the project file. Saving login details will allow persistence between project instances. If this box is left unchecked, the username and password reentry will be required on project load.

Find Data Servers

To find available data servers, click the 'Find' button near the top of the screen beside the 'Catalog Type Options' drop-down. This will search for data servers within the network. If you now the name of your data server, simply enter it in the 'Data Server' field. Please note that this functionality only applies to MS SQL Servers.

Connect to Database

Once you have selected a Data Server and entered your User Name and Password, click the connect button. Ouiji will then attempt to connect to the selected Data Server. If successful, a list of data catalogs will populate the 'Select Catalog' drop-down. Select the desired catalog.

Add Catalog Entry

After completing the catalog set-up and selecting a catalog, click the 'Add' button at near the top of right of the panel. Ouiji will test the connection and add a new catalog entry to the project. Catalog entries will be displayed in the panel on the right side of the screen. To edit existing entries, simply click on the desired entry, modify the properties and click the 'Update' button.

Data Source Categories

The Data Source screen provides the option for two separate Data Source Categories:

1. Primary Data: The Primary Data Source Category defines variable data sources which will be pulled from live input sources at export run time. Primary data cannot be merged with data from other primary data sources.

2. Secondary Data: Secondary data is imported into the Ouiji project and is persistent. This data can be mapped to Primary Data fields on the 'Mapping Data Sources' screen and merged at export run time. Secondary data may be re-imported, should data within the source data change.

Data Source Types

Ouiji currently supports four input data source types: CSV, Database Catalog, XML File and Excel Files.

Data Source Type: CSV File

The Ouiji CSV Files (Character Separated Value) data source type allows for the import, parsing and mapping of character delimited files and are parsed based on the supplied delimiters. Most CSV files will contain only one delimiter, however Ouji can parse on secondary or 'sub' delimiter if supplied.

To create a Data Source Definition from CSV:

1. Give the Data Source Definition a Name
2. Select a sample Source File
3. Provide a delimiter
4. Provide a secondary delimiter (optional)
5. Provide a clip data string (optional)
6. Provide string replacements (optional)
7. Select whether or not to ignore the first line
8. Click the Add button at the top of the pane

Once you click the 'Add' button, Ouiji will create a list of target 'Data Column Definitions'. These definitions will allow you to map specific columns to tags within defined 'Export Templates'. Target Column Definitions can be modified by clicking on them in the tree view nodes under the Data Source definition.

Primary Delimiter: A primary delimiter is required to parse a CSV file. Target Data Column definitions (Column Data Name and Label) will be extracted from the header line of the file. If there is no header line in the source file, the Target Data Column name and label can be renamed manually. Common delimiters include: the comma ( , ), semi-colon ( ; ) and pipe ( | ).

For example, if the primary delimiter of the source file is a pipe ( | ) and the header of the source file looks like:

>> Id | Name | Age | Title

Ouiji will create Target Data Columns of Id, Name, Age and Title, which can be mapped to output templates.

Secondary Delimiter: The secondary or 'sub' delimiter is used to parse lists or sub-data inside the Primary delimited data.

For example, if the primary delimiter of the source file is a pipe ( | ), the secondary is a comma ( , ) and the header of the source looks like:

>> Id | Name , First , Last | Age | Title

Ouiji will create Target Data Columns of Id, Name.First, Name.Last, Age and Title.

Clip Data: The Clip Data field allows you to replace a part of the source data with and empty string. There are two clip data tags available {Header} and {LABEL}. These are related to the 'Target Data Column' and the 'Mapping Alias' of the Target Data Columns respectively.

For example, if the primary delimiter of the source file is a pipe ( | ), the secondary is a comma ( , ) and the source looks like:

>> Id | Name , First , Last | Age | Title
>> 0900 | First=Charles , Last=Emmerson | 42 | Manager

and your clip data is {LABEL}= the post-parse data will be

>> 0900 | Charles , Emmerson | 42 | Manager

String Replacements: String replacements will replace all matching strings in the source data with a paired replacement string.

Data Source Type: Database Catalog

The Ouiji Database Catalog data source type allows for the import and mapping of data with a query against a database catalog as defined in the 'Define Data Catalog' Screen.

To create a Data Source Definition from a Database Catalog:

1. Give the Data Source Definition a Name
2. Select a database catalog (define on the 'Add Database Catalogs' screen)
3. Supply a query (** Note that queries must contain aliases to populate 'Query Fields')
4. Adjust Query Fields (if required)
5. Provide string replacements (optional)
6. Click the Add button at the top of the pane

Once you click the 'Add' button, Ouiji will create a list of target 'Data Column Definitions' based on the entries in the 'Query Fields' field. These definitions will allow you to map specific columns to tags within defined 'Export Templates'. Target Column Definitions can be modified by clicking on them in the tree view nodes under the Data Source definition.

String Replacements: String replacements will replace all matching strings in the source data with a paired replacement string.

Data Source Type: XML File

The Ouiji XML File data source type allows for the import and mapping of data in an XML file.

To create a Data Source Definition from a Database Catalog:

1. Give the Data Source Definition a Name
2. Select a sample Source File
3. Provide an XML Namespace
4. Define and XPath and XPath Alias (optional)
5. Define Attribute Identifiers (optional)
6. Provide string replacements (optional)
7. Click the Add button at the top of the pane

Once you click the 'Add' button, Ouiji will create a list of target 'Data Column Definitions' based on the entries in the XML nodes and / or Attribute Identifiers. These definitions will allow you to map specific xml nodes to tags within defined 'Export Templates'. Target Column Definitions can be modified by clicking on them in the tree view nodes under the Data Source definition.

String Replacements: String replacements will replace all matching strings in the source data with a paired replacement string.

Data Source Type: Excel File

The Ouiji Excel File data source type allows for the import and mapping of data contained within Excel workbooks.

Creating & Editing Templates

The template screen contains two fields:

1. Name:  Requires a unique name which Ouiji will refer to when referencing the template entry.

2. Template Format: Clicking on the template format 'Edit' button will open the Template Editor.

Template Editor

The template editor enables developers to easily design custom data export formats. The editor has five major components:

1. Editor Window:  The Editor Window allows developers to create and edit their output template formats. Clicking on a line in the Editor Window will load the selected line into 'Line Editor' found directly below the Editor Window.

2. Line Editor: This Line Editor is used to modify the line currently focused on in the 'Editor Window'. To commit the line changes from the Line Editor to the Editor Window, click the 'OK' button to the right of the Line Editor field. The Line Editor also allows developers to add or edit 'Ouiji Tags'. Ouiji Tags are used to map input data to templates. To add a Ouiji Tag, click on the 'New Ouiji Tag' tab at the top of the editor. On confirmation of the tag set-up, the tag will be inserted at the current caret position within the Line Editor.

3. Ouiji Tag Editor: When a line is loaded into the Line Editor, it will check for 'Ouiji Tags'. If it finds properly formatted Ouiji Tags it will add the tags to the tab bar above the Line Editor. Clicking on these tabs will show the selected Ouiji Tag properties. If modifications are made to the selected Ouiji Tag, they will be reflected in the Line Editor on confirmation.

4. Snippets: The right panel of the editor contains a treeview. This treeview will display generic and specific text and tag snippets based on the 'Options' dropdown to help the developer assemble their template. Specifically for 'OpenText Import' option lists all the tags available for the creation of OpenText Import files.

To add a snippet to the Editor, simply select the desired snippet and click the 'Add' button. This will add the formatted snippet to Editor at the currently selected caret position.

4. Confirmation: Once template modifications have been completed, click the 'OK' button at the bottom of the Template Editor. To cancel modifications, click the 'Cancel' button. This will return you to the 'Create Export Templates' screen. NOTE: This template will not be saved until the 'Add' or 'Update' buttons are pressed on the 'Create Export Templates' screen.

IMPORTANT: Modifying 'Ouiji Templates' already mapped to 'Data Sources' may break data mapping. In this case, the remapping will be required on the 'Map Data Sources' screen.

Ouiji Mapping Tag

The Ouiji Mapping Tag, referred to a the 'Ouiji Tag' is the guide to the data mapping process. It can be typed directly into the Editor Window or modified in the Line Editor. It is recommended using the Line editor for new users of the system.

While there are a number of options for how the tag should handle mapped data, the only two properties required by a Ouiji tag are the 'Label' property and the 'Id'.

Tag Id: The tag 'Id' must be a unique identifier. This Id is referred to by the system during the export process.

Tag Label: The tag 'Label' will be displayed to the developer during the 'Data Mapping' process. It is recommended that the Label property be explicit and related to the input data to simplify the mapping process. The 'Mapping Data Sources' screen has the ability to auto-match input 'Data Source' fields to Ouiji Tags found in templates if the two have similar Labels or Ids.

Ouiji Tag Format: [ouiji.field id='{UNIQUE ID}' label='{READABLE LABEL}' /]

Ouiji Tag Example: [ouiji.field id='mediatype' label='Media Type' /]

To add a new Ouiji tag using the Line Editor:

1. Place Caret:  Place the caret in the Line Editor at the desired insertion position.

2. Add New Tag:  Click on the 'New Ouiji Tag' tab found above the Line Editor. This will reveal the tag properties sub-form.

3. Fill in Properties:  Provide an unique Id, Label and other desired properties.

4. Confirmation:  Click the 'OK' button next to the tag properties sub-form. The tag will then be inserted into the line editor at the selected caret position. A new tab will in the tab bar above the Line Editor related to the newly inserted tag.

5. Editing Existing Tags: Click on the tab above the Line Editor with the related Ouiji Tag label.

Ouji Data Looping

Each template must contain a Ouiji Data Loop start and end tag. Any content outside of the data loop will be output as it appears in the template. Any content inside the data loop will be output for each record contained within the mapped input 'Data Source' data set. The data loop instruction is comprised of two tags:

1. Start Loop:  The Start Loop tag must be placed before the desired start of the data looping position.

Format: <!--ouiji.startloop-->

2. End Loop:  The Start Loop tag must be placed after the desired end of the data looping position.

Format: <!--ouiji.endloop-->

To add a data loop to the template, either enter the tags manually or, more simply, select the 'Data Loop' element in the 'Snippets' window and click 'Add'. This will insert the data loop tags at the selected caret position within the Editor Window.

Creating a Data Map

To create a 'Ouiji Data Map', an 'Output Template' is paired with a 'Data Source' definition. To facilitate this process, the 'Map Data Source Screen' has two distinct parts:

1. The Editor Area:  The editor area on the left side of the screen is contextual and will change depending on your current action from either editing the 'Data Map' to mapping the 'Template Tags' with 'Data Source' fields.

2. Template Data Maps: The panel on the right side of the screen contains a tree view of existing data maps. The top nodes of the tree view represent the existing data maps. The child nodes of the tree view represent the tags found within the selected Output Templates.

Clicking on the top nodes of the tree view will display Data Map options. Clicking on child nodes will display Template Tag mapping options.

The 'Data Map' form has three parts:

1. Name: Requires a unique name which Ouiji will refer to when referencing the Data Map.

2. Template: The template drop-down will contain all templates within the current Ouiji Project.

3. Primary Data Source: The Data Source drop-down will contain all Primary Data Sources within the current Ouiji Project.

Once a Name has been entered and a Template has been paired with a Primary Data Source, press the 'Add' button at the top of the screen. A new top level entry will appear in the 'Template Data Maps' tree view, its children populated with the Ouiji Tags found inside the selected Templates data loop.

Mapping Template Tags

Once an 'Output Template' has been paired with a 'Data Source' definition, and added to the 'Template Data Maps' panel, all 'Ouiji Tags' found in the selected template's data loop will be added as children to the entry. To map theses tags to data source definitions fields or columns, click on the desired tag definition in the tree view. Clicking on a child node will display the node mapping screen in the editor area. The tag can then be mapped to the desired field in the 'Primary Data Source'. This screen consists of four parts:

Important: This process is much simpler than it sounds. It's just a matter of clicking on a node in the tree view, and auto-matching or selecting the related fields in the source data.

1. Source Fields: The source field drop-down will list all data fields or columns found in the Primary Data Source data set definition. If the input field has a similar name to the selected Ouiji Tag label, the developer can attempt an auto-match by clicking the 'Match' button.

2. Merge Source (Optional): The Merge Source drop-down will be populated with all 'Secondary Data Sources' found within the current project. If the data in the 'Primary Data Source' record should be replace data from a 'Secondary Data Source', select which data source should be used in the merger.

3. Merge Comparison Field: The Merge Comparison Field will list all fields or columns found within the selected Merge Source. During the export process, data within the selected Primary Data Source record field will be compared to data within the Merge Comparison records. If a match is found between the Primary and Merge Comparison data, the Primary Data will be replaced with data found in the 'Merge Value Fields' data field as described below.

4. Merge Value Fields: The Merge Value Field will list all fields or columns found within the selected Merge Source. During export, should data from the Primary Source record field match data found in the Merge Comparison Field, the Merge Value Field data will replace the existing data within the Primary Data Source record.

Defining Data Exports

Once a data map has been created, it can be mapped to an input directory. Ouiji will search this directory for files matching the 'Export Definitions' file format. Each Export Definition screen requires the following:

1. Name:  Requires a unique name which Ouiji will refer to when referencing the entry.

2. Input Directory: The input directory defines the path where the source data (data requiring conversion) will be located.

2a. Input Directory File Format: The file format tells Ouiji which the format of the files to process within the selected input directory.

3. Output Directory: The output directory defines the path where the converted files will be output to.

3a. Input Directory File Format: The file format defines the name format of files generated by Ouiji to the output directory.

4. Records Per File: Indicates how many records should be processed and exported per output file. If the number is set to zero, the number of output records will equal the number of input records.

5. Range Start / End: Range Start / End only applies when querying directly against a database. This will allow developers to run a subset of the query defined in the associated Data Map Data Source.

Summary Panel

The Summary Panel on the left side of the screen displays all configurations and settings within the current Ouiji Project. The right side of the screen contains a project history as well as a processing queue.

Exporting Data

Once the developer defined 'Data Exports', they will show up in the Summary Panel, indicated by a blue bullseye icon. To run the export, simply click select the desire Data Export and click the 'Run Task' button which will appear above the panel. Multiple tasks can be run simultaneously. These tasks will appear in the Export Queue in the right panel.

Export History

On 'Data Export' completion, a new entry will show up in the 'Export History' panel to the right of the screen. If any errors occurred during the conversion and export process, the entry will be indicated by an alert icon.

Detailed History

To review the details of a Data Export completion, click on the desired entry and press the 'View History' button found at the top of the panel. This will open up the History Viewer window. On the right side of the history viewer is a tree view containing past export events. Click on the children of these events to see details about the event listed in a data grid on the left side of the screen. To get further details, click on the cells of the data grid and their contents will be listed in the panel below the data grid.