Connector Repository 3.3.1

Index Data

November 2013

Description:

The Connector Repository is an administrative tool used to manage all connectors available for use, development or enhancement via Index Data’s Connector Platform.

The repository presents a single, unified interface for all connectors, including those in production, in development, or not currently in use. It also features automated testing for all existing connectors, as well as an integrated display of reliability statistics drawn from actual production usage.

Creating & Managing Connectors in the Repository:

Public vs. Private Repositories

Access to a customer-specific, "private" repository is controlled by login credentials; each customer repository also includes full access to the “public” or "global" repository , where recent versions of connectors created by all CF users reside.

Although customers create, edit, and save connectors in their private repository, unless blocked, all connectors from each private customer repository uploads to "public" once it is released to production.

Creating & Editing Connectors

Additionally, connectors in the public repository are available to all private repos. Thus, before creating a new connector from scratch it is advisable to search the public repo to see if it has already been constructed by another community member. If the desired connector can be located in the public repo, simply import it to the private repository (via the "Localcopy" link in the Detail view) to use as is, to edit, or to alter in some specific fashion via the Connector Builder.

To save changes made to connectors in the Builder, either save the file locally, or upload it to the home repository. By default, connectors are uploaded to the private repository assigned to the active login--never directly to the public repo.

Each time a connector is uploaded it is saved with a new version number appended to the file name. Each new version number increases by one from the the highest number in either the private or the public repository for the given connector. For example, the connector "doe_patents.14.cf" indicates that the current version is the 14th.

Note: Since many private repos contribute to the public one, it is possible that a public repo connector may have a higher version number than a given specific private repo, depending upon which repo user edited it last. For instance repoA user may have saved "myconnector.5.cf" to public, after which repoB user downloaded version 5 from public, edited it as needed, then saved and released "myconnector.6.cf" to public. Consequently, repoA's version would still be 5, although the latest version in the public repo would be 6.

When releasing significantly different versions to the public repo, it is best to rename the connector and include notes about its content in the metadata.

Testing Connectors

Although testing of new or edited connectors is often performed in the Builder, the repository also provides advanced test functions. To run a full suite of diagnostic tests from the repo simply click on the "Test" link in the "Action" column of the connector's Detail view.

An interactive result page for a successful test resembles the following:

Note that all of the lines in dark blue text represent interactive links that display diagnostics and/or log files. For example, clicking "Messages: brief" for the first test shown above displays:

query: {"title":"mechanism"}
First page record count: 25
Second page record count: 25
3rd page record count: 25

In the event that a connector fails the scripted tests, this screen will not only indicate that failure, but its contents can be used to diagnose and fix the problem.

Test results are always reflected in the connector Detail View, as shown below. Note that while version 14 is "OK", version 13 had "Failed."

Note: "Test" results are not the same as "Reliability Statistics." Tests such as those described above that have been run from a specific repository against a specific engine -- usually (but not necessarily) that of the production site -- are recorded as shown above.

Reliability performance data, however, displays in the repo List View and is derived from invocations of the connector from a production site, not via the repo. Thus, reliability statistics reflect the live use of a connector by MasterKey or other UIs calling the connector into use as a metasearch target.

Releasing Connectors to Production

After creating or editing a connector, and after it has been successfully tested, click "Release" in the "Action" column of the connector's Detail View to put it into production.

Not that an "unreleased" connector will still upload to the public repository unless blocked from doing so in the Detail View.

---

Repo List View:

The repo list view includes navigational features, current status, and reliability statistics for each connector.

Repo identification and statistics:

Below the Index Data banner, note the current repository identification (name) and summary statistics, including total connector count, as well as "unstable" and "failed" counts.

Sorting:

The connector list view can be sorted in ascending order by clicking on column headers: Title, File Name, Latest versions and Reliability

Column contents and functionality:

Title & File name
: "Title" and "File name" display from the CF Builder's metadata dialog box
: Click on a title to open the connector's Detail view.

Latest Versions column contents
: Displays the numeric designation of the version of the connector, followed by a functional descriptor in parenthesis:
3 (latest) denotes the most recent version of a connector--in this case version 3;
3 (production) indicates version 3 is currently used in a private production repo (Note: "production" is visible only in customer-specific repos, not in the public repo);
3 (latest & production) means a single connector version fits both categories;
3 (latest), 2 (production) means that version 2 is in production, but a newer version of the same connector (3) exists.

Latest Versions column features:
: Hover over "latest" or "production" to display the connector's last upload date (Note: Sorting for this column is on the last upload date)

Click "latest" or "production" to open that specific connector version in the CF Builder

The color of the version number indicates its current status:

green indicates a connector is fully functional
red indicates failure for the last three days
orange means the connector failed in the current day, but worked during the prior two days

Reliability:
: Displays connector's production success rate as a percentage and as raw statistics

Hover over data displays the date the last automated test was run on a production connector

Click on data in reliability column to display details and diagnostics, including time stamp, diagnostic error and the relevant search query.

##Footer information and functionality:

In the lower left of the List View screen, "Available repos:" enables navigation between a private repo and the public one.

The "Logout" link clears the current login.

---

Repo Detail View:

Header information

: Below the Index Data banner, note the header that identifies the connector and the current repo

The box immediately below the header displays the current production version; clicking on "Remove from Production" does just that. This feature allows replacement of current production connectors with later edits.

Title and Author display from metadata in the CF Builder

Home Page refers to the web interface referenced by the connector and features a direct link

"Settings" should remain "public" unless there is a compelling reason to prevent upload to the public repo. Note that even "unreleased" connectors upload to the public repo unless blocked using this dialog.

Version Details

The Detail view contains data relevant to all versions of a connector, including file name and version, date of last modification and optimization, reliability statistics (for production connectors), and the most current testing status. The last column ("Action") allows for testing, deletion, optimization and release (to production) for each version. Note that once a connector has been released that option is no longer available for the given version--unless it is subsequently removed from production.