Matrix Store v2.x

Before you start

In order to use this MatrixStore archiving plugin, you need:

CatDV Server 8.0.7p3 or later OR 9.0.1p14 or later

CatDV Pegasus or Enterprise Client 13.0.5 or later

CatDV Plugin licence 'ARCMS' (Rest API licence with multiple sessions)

Client set to calculate MD5 when importing files, ideally:

Client -> Preferences -> Import

Check "Automatically calculate MD5 when importing files"

NB This will make imports slower. If this is not done, the plugin will calculate its own MD5 each time a file is archived, which will slow down the archiving process.

To trigger MatrixStore file transfers from the Worker you need:

MatrixStore Worker Plugin 2.2.8, included in this installation as MatrixStoreWorker.catdv

In addition, to run the archive service standalone (outside the server):

CatDV Service Control Panel 1.3.1 or later

New Installation

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on Unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Set server properties required by the plugin:

Open 'Server Config' from the control panel

Enter required properties into the CatDV Server tab, 'Other' field. See Plugin server properties below.

Restart the CatDV server from the control panel

Open the CatDV client (or log the client back in to the server)

Configure one or more service accounts. The first account will always be the default account, used to verify that MatrixStore is available and used as a fallback when necessary. See Manage service command below for information on the details and settings for a service account:

In the client, run Tools->Manage MatrixStore Archive Service

On the accounts tab enter the service account details. Mandatory details are flagged with an asterisk. Note that once a service account has been used to archive a clip, it will no longer be possible to update its identifier.

Click 'Add'.

Set up M edia S tore S ervice Path M appings to automatically map drives/folders to archive locations. If you plan to always use a single service account and a single plan, you may skip this step and use default location mapping. However, it is still recommended to use this approach if you don’t want the file ‘paths’ on MatrixStore to include the path of the storage volume.

Optionally configure archiving, processing and UI settings for the plugin via the corresponding tabs on Tools->Manage MatrixStore Archive Service.

If R unning A rchive S ervice S tandalone, then configure and start the archive service via the service control panel.

Verify the service setup: In the client, run Tools->Manage MatrixStore Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if MatrixStore is not currently accessible.

If the MatrixStore worker plugin IS NOT being installed:

Delete MatrixStoreWorker.catdv from the plugin directory (see first step).

If the MatrixStore worker plugin IS being installed:

Move MatrixStoreWorker.catdv FROM the plugin directory installed in the first step TO the worker plugins directory:

e.g. on a Mac: /Library/Application Support/Square Box/Extensions

e.g. on Windows: %ProgramData%\Square Box\Extensions

e.g. on Linux: /usr/local/catdvWorker/extensions

If the worker is running, re-connect to the server (Edit Config / CatDV Server / Reconnect) so that the archive plugin fields are available.

Verify that the MatrixStore worker actions are available. It may be necessary to quit and re-open the worker to pick up the new worker actions.

Upgrade Installation

Make a note of the current plugin version from this document (or see latest version in Release Notes).

If running the archive service standalone, use the service control panel to stop the service.

Stop the CatDV server from the control panel

Copy the whole directory extracted from this zip to the plugins directory:

e.g. on Unix systems: /usr/local/catdvServer/plugins

e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins

Remove or move the directory / files for the prior plugin from the plugins directory.

Carry out any 'Upgrade' instructions for each plugin version listed in the Release Notes above the last installed version, working back from the oldest applicable version

Restart the CatDV server from the control panel

Read through the 'Changes' for each plugin version listed in the Release Notes above the last installed version, and go to Tools -> Manage MatrixStore Archive Service to verify that the details / settings for each account and the archiving / processing / UI settings for the plugin are correct for this installation. See Manage service command for more details.

If R unning archive service standalone, then update and start the archive service via the service control panel.

Open the CatDV client (or log the client back in to the server)

Update (or delete) the MatrixStore worker plugin by following the instructions for the worker plugin in a new installation.

Plugin server properties

When running the archive service "InPlugin", the following server properties must be set:

catdv.matrixstore_archive.licence_code = <generated licence code for plugin>

When running the archive service standalone, the properties for the archive service must include:

catdv.matrixstore_archive.licence_code = <generated licence code for plugin>

catdv.rest_api.host = <CatDV server hostname>

catdv.rest_api.port = <CatDV API port>

catdv.rest_api.user = <CatDV API user>

catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev)

Typical rest API values for a development machine would be (password property omitted for a blank password):

catdv.rest_api.host = localhost

catdv.rest_api.port = 8080

catdv.rest_api.user = Administrator

In addition, the following optional property may be set to turn on debug logging for the plugin:

catdv.matrixstore_archive.debug = true

In addition, the following optional property may be set to change the log level for the MatrixStore SDK:

catdv.matrixstore_archive.ms_log_level = FINE

Valid values are SEVERE (least output), WARNING, INFO, CONFIG, FINE, FINER, FINEST (most output). The default is WARNING.

Running archive service standalone

By default, the service that handles archive jobs runs inside the CatDV service. The following is required to run the plugin's archive service on a separate machine from the CatDV server:

CatDV Service Control Panel 1.3 or later

To configure and start the standalone archive service using the service control panel:

Open the Manage service command in the client / web plugin UI

Verify that at least one service account has been configured

In the Processing tab, set the ‘Service mode’ to ‘standalone’ and click ‘Save’

Re-open the Manage Service Command and verify that the service is ‘Stopped’. If not, restart the CatDV server.

Install the CatDV Service Control Panel

Open the CatDV Service Control Panel

Click 'Service Config' to open the service configuration

Fill in license details on the Licensing tab

Optionally, fill in details of a REST endpoint on the REST Service tab

Fill in server connection details on the Server Connection tab (connection can be checked on the control panel)

Enter required service properties on the Settings tab. See Plugin server properties. (**Note the server connection details are duplicated here for now, pending a plugin update to remove them)

On the Installation tab, note the install location ('Server Install Path')

On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.

Click 'Start' to start the archive service.

To update the standalone archive service:

The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.

Open the CatDV Service Control Panel

Add / edit service properties on the Settings tab if applicable.

Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.

Click 'Start' to start the archive service.

Media Store Service Path Mappings

Media Stores should be used to automatically map file storage to archive locations by adding a service path to each media store. The service path determines the service account, vault and (optional) folder path for the archived files. Service accounts are set up via Tools -> Manage MatrixStore Archive Service (see Managing Service Accounts) and are given an identifier when they are created.

For MatrixStore a service path has the format:

For example, if a Media Store were set up with the following two entries:

/Volumes/dance-videos/drafts

matrixstore://squarebox-test/demo/dance

Then a clip located at:

/Volumes/dance-videos/drafts/greatest-show/this-is-me

Would be archived:

using the credentials from the 'squarebox-test' account

into the vault with name ‘demo’

with a path of '/dance/greatest-show/this-is-me'

Each Media Store is intended to represent a distinct segment of physical storage so, to avoid ambiguity, no path should be present in multiple media stores and paths should not overlap (e.g. there should not co-exist media stores for /Volumes/dance-videos/drafts and /Volumes/dance-videos/drafts/special).

If you want to avoid having duplicate index entries for files with the same path and name from different drives, make sure they are mapped to either distinct vaults or distinct folders within the same vault.

The 'folder path' can be set or overridden on archive if 'location' is set as an allowed override. See Manage Service Command (UI TAB).

Restore path mapping

If a restore location is not writable when a restore job is run, the plugin will use Media Stores to attempt to find an alternative writable restore location. If there is a Media Store with a path matching the restore location, the plugin will check whether any of the other paths in that Media Store are writable and substitute the root directory of the restore location accordingly.

The restore location can be overridden when a restore job is scheduled, if the option to override it is turned on. In earlier versions of the plugin, in this case no file hierarchy was preserved – all files were written directly to the specified directory. Now the plugin will use Media Stores to try and preserve file hierarchy for restored files. If any Media Store path is found that matches the default restore location, it will be used to substitute the root directory of the default restore location with the override.

Note that if many files are restored together to an overridden location, some files may have hierarchy preserved and some may not, depending on whether matching Media Store paths are found.

Using the plugin

The plugin comprises various commands that are available from the Tools menu in the client.

File transfers are initiated by the schedule copy / move / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during / after transfer.

The plugin includes the following commands:

Command	Description
Manage MatrixStore Archive Service	View full status details of the archive service and manage the service account(s) required to access MatrixStore archives. Service account details include account name, account key and default container. If a connection to MatrixStore cannot be made an error is displayed and the values are not saved
View MatrixStore Archive Service Job Queue	Lists jobs, which can be filtered by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs.
Schedule copy to MatrixStore	Adds a copy job for each clip selected (if multiple clips reference the same source media only one job is created). Copy jobs can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and MatrixStore is accessible). When the copy job is run, the source media associated with the clip is copied from the file system to MatrixStore and the original file is preserved. Note that if a file is copied to MatrixStore multiple times with the same archive path/location, MatrixStore will store multiple copies. The plugin will always reference the latest copy via it’s mxs object id.
Schedule move to MatrixStore	As copy but on successful transfer of the file to storage the job attempts to delete the original file. If for some reason the deletion fails this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the job data. Note that if a file is moved to MatrixStore multiple times with the same archive path/location, MatrixStore will store multiple copies. The plugin will always reference the latest copy via it’s mxs object id.
Schedule restore from MatrixStore	Adds a restore job for each clip selected (if multiple clips reference the same source media only one job is created). Restores can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and MatrixStore is accessible). When the restore job is run, the source media associated with the clip is copied from MatrixStore to the file system.
Purge files archived to MatrixStore	Deletes the source media associated with the selected clips if they have been successfully archived.

Managing Service Accounts

The accounts tab on Tools->Manage MatrixStore Archive Service provides facilities to manage the service account(s) used to connect to MatrixStore archives. A single account is usually sufficient to operate the plugin. However, if required, there is an add-on which enables multiple accounts to be set up for security purposes etc. This is enabled as an add-on via a special server config property.

When there are multiple accounts, the account used may be determined automatically via Media Store Service Path Mappings or selected by the user on archive / restore if allow override of Account ID is turned on from the UI tab of the M anage service command.

An account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes cannot be made to an account such that files can no longer be restored. Note it is not possible to change the account identifier on an in-use account.

The accounts tab provides the following operations:

Button	Description
Clear	Clears the current selection / details so that only default values are filled in.
Set Default	Updates the default account to the current selected account.
Add	Creates a new account with the specified details.
Unlock	Unlocks an in-use account in preparation for update / delete.
Update	Updates the selected account with the specified details. If an account is in use, it must be unlocked before it can be updated. Please take care when updating an account which is in use, as changing some settings could break the restore of files archived with that account. For example, if the account key is switched to one which does not have the same access or if any encryption settings are changed. Note it is not possible to change the account identifier on an in-use account.
Delete	Deletes the selected account. Please take care when deleting an account which is in use, as it will no longer be possible to restore any clips arched with that account. This feature is intended for removing test accounts and any applicable clips should be deleted or restored and re-archived.

Manage service command

The following can be configured from Tools->Manage MatrixStore Archive Service in the web or desktop UI for the plugin:

ACCOUNTS TAB / DETAILS (for an account):
Field	Description
Account Identifier	Identifying name for this service account. This may contain only alpha-numerics and hyphens. Note it is not possible to change the account identifier on an in-use account.
Cluster Host(s)	Cluster host addresses for connecting to this service account. If there are multiple addresses, they should be separated with a comma.
Auth Type	Indicates whether to authenticate via a username and password or an access key (ID and secret). The two fields below must be filled in accordingly.
Username / Access Key ID	Username (or Access Key ID) for connecting to this service account
Password / Access Key Secret	Password (or Access Key secret) for connecting to this service account
Default Vault	The name of the vault that will be used as the archive fallback for this account by default. This is not applicable when a media store service mapping applies. Otherwise, if the UI tab settings allow the user to override the vault then this will only be used as the default the first time the user does an archive. Subsequent archives by the same user will default to the last value they entered.
ARCHIVING TAB:
Exclude archived files from archive	Skip files that have already been archived when scheduling jobs to copy / move to archive, unless they currently have a copy / move failed status. Default is “yes” for new installs. Note, if the file(s) were previously copied, an attempt to move the file will simply be skipped, it will not purge the file. When this option is turned on, use the purge operation to purge files which have already been copied. This option can be temporarily disabled if it is necessary to resubmit files for some reason.
Exclude existing files from restore	Skip files that already exist when scheduling restore jobs. Default is “yes” for new installs.
Purge directories	Determines whether or not empty directories are deleted when purging (or moving) files.
PROCESSING TAB:
Service mode	NB Always ensure that the standalone service is NOT running prior to changing this value. Determines whether the job processing service runs in-server (as a thread in the CatDV Server) or standalone (via the Service Control Panel). The initial/default value is in-server. When this value is switched, the plugin will attempt to start or stop the in-server service as appropriate. If this does not appear to have been successful, as reported on the Overview tab, then it will be necessary to restart the CatDV server to apply the change.
Concurrent job limit	Determines the number of transfers that the archive service will attempt to run concurrently. The default value is 4.
Number Delayed retries	The number of times a waiting job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service.
Maximum retry delay	Limits the maximum duration between retries of waiting jobs. The default value is blank.
Loop	Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. The default value is 30s
Retry running job delay	Determines the duration after which a Job which is running will be restarted if it has not been updated during that period. Default is 1hr
Stalled job delay	Determines the duration since a running job's last progress update, after which its’ status is flagged as stalled. Default is 1m.
UI TAB:
Restrict command access	Restricts the specified plugin commands to sys admin users. Can be used to hide 'config' commands only (i.e. Manage Service), ‘config / archive’ (i.e. to restrict copy and move to archive but not restore and purge), ‘config / archive / purge’ (i.e. to restrict copy, move and purge but not restore) or 'all' commands. Note that a server restart is required to ensure that this change is picked up under all circumstances. Default is ‘none’.
Days to display	The number of days into the past for which jobs are listed in the job queue. Any job updated in this time period is listed. The default value is 10.
Max no jobs to display	The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000.
Allow overrides	Enables the facility for users to override one or more parameters at the point of archive or restore.

**Any duration can be set in h (hours), m (minutes), s (seconds) or any combination in that order, e.g. 2h, 2h 30m, 5m, 1m 30s, 1s.

Archive details pane

The plugin automatically creates a panel entitled "MatrixStore Archive" containing the clip metadata which describes its MatrixStore archive state, including:

Command	Description
squarebox.catdv.archive.MatrixStore.serviceType	Type of service responsible for file transfer
squarebox.catdv.archive.MatrixStore.serviceName	Name of service responsible for file transfer
squarebox.catdv.archive.MatrixStore.status	Archive status
squarebox.catdv.archive.MatrixStore.location	Location of file in storage
squarebox.catdv.archive.MatrixStore.restoreLocation	Location file will be (if job pending) or was last restored to
squarebox.catdv.archive.MatrixStore.date	Date (timestamp) of latest change in archive status
squarebox.catdv.archive.MatrixStore.dateLastArchived	Date last archived
squarebox.catdv.archive.MatrixStore.dateLastRestored	Date last restored
squarebox.catdv.archive.MatrixStore.numArchives	The number of times the clip has been successfully archived
squarebox.catdv.archive.MatrixStore.archiveKey	Identifier of file in storage
squarebox.catdv.archive.MatrixStore.batchID	Identifies the batch of files with which the clip was archived
squarebox.catdv.archive.MatrixStore.jobID	Identifier of current / latest archive job
squarebox.catdv.archive.MatrixStore.parentJobID	n/a for MatrixStore (related to bulk archives)
squarebox.catdv.archive.MatrixStore.userId	ID of user initiating current / latest transfer
squarebox.catdv.archive.MatrixStore.historyJson	Record of all archive activity in json format
squarebox.catdv.archive.MatrixStore.history	Record of all archive activity (including before historyJson added)
squarebox.catdv.archive.MatrixStore.purged	Indicates whether or not file has been purged by plugin (reset on restore)
squarebox.catdv.archive.MatrixStore.purgeError	Details of purge failure
squarebox.catdv.archive.MatrixStore.accountIdentifier	Identifier of the MatrixStore Storage Account in CatDV
squarebox.catdv.archive.MatrixStore.vault	Name of vault to transfer file to / from
squarebox.catdv.archive.MatrixStore.mxsObjectID	Object ID uniquely identifying a copy of the file in MatrixStore

Known Issues

n/a

Licence Code

IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 8 license agreement.

Square Box Systems Ltd.

May 2021

Release notes

2.2.9 (2021-05-24)

Upgrade

- server to 9.3.3

Changes

- Add ‘No to all’ option for purge clip confirmation

- Fix for plugin license expiration causing ballooning logs

- Modify handling of waiting jobs to limit the resources they consume, including throttling re-queue of waiting jobs.

2.2.8 (2020-10-08)

Upgrade

- If using worker, update worker plugin to 2.2.8 (included with this plugin)

- If there is any script in place which relies on the value of the cross-plugin source media ‘archiveStatus’ field (e.g. for archive status traffic light indicators), ensure it is updated to reflect that ‘Archived’ may now be either ‘Archived’ or ‘Purged’.

Changes

- Add service config option to ‘Exclude archived files from archive’, now the default for new installs. Where files are not expected to change, this ensures there is no duplication of archive time / cost if files are picked up repeatedly.

- Add ‘purged’ flag to archive parameters

- Update generation of clip level archive status to include a ‘purged’ state

- Improve consistency of error reporting between server and worker plugins

- In worker plugin, return success if file(s) have a pending job of the same type, to reduce errors from picking up duplicate files

2.2.7p2 (2020-07-29)

- Add support for account authentication by either username/password OR access key id/secret, for MatrixStore cloud offering

2.2.7p1 (2020-06-29)

Upgrade

- Server to 8.0.7p3 or later OR 9.0.1p14 or later

Changes

- Fix for conflicting clip edits in worker archive plugin actions, following meta-clip status update

- Fix to try and ensure that duplicate clips are excluded from restores, based on media path rather than file path

2.2.7 (2020-05-29)

Upgrade

- If using worker, update worker plugin to 2.2.2 (included with this plugin)

- If running service standalone, update CatDV Service Control Panel to 1.3.1

Changes

- Update summary archive status on meta-clips when meta-clip member archive status changes

- Additional manage service option to restrict access to purge command along with archive commands

- Add option to exclude existing files from restore jobs

- Update log of archive history to include location restored to, rather than location restored from

- Ignore trailing slashes in media store paths

- Fix for purge of restored file where mapped media path has changed since file was archived

- Fix default html format for json history field, to resolve issue of archive pane flipping / redrawing

- Fix purge of empty complex media folders from bulk jobs

- Fix for issue restoring files ingested with non-windows paths on Windows.

- Fix field type for archive fields ‘media file path’ and ‘last archived params’

2.2.6p2 (2020-02-26)

- fix for duplicate folders on MatrixStore

- eliminate ‘\’ from MXFS_PATH for folders

2.2.6p1 (2020-02-24)

- update worker to 2.2.1 to fix worker purge action and ensure it doesn’t apply for bulk queries

- fix worker plugin commands so that they don’t update the allowed overrides settings for the service to all be enabled

- ensure that assets linked back from MatrixStore, which have no custom ‘file_orig_path’ field, can be purged after restore

- ignore trailing slashes in media store paths

2.2.6 (2020-02-11)

Upgrade (whilst CatDV server is down)

- backup the service table from the DB if it contains multiple rows

- run catdvmatrixstoreplugin2.2.6.sql against the CatDV DB to update the service UID to the new combined value

**Note that if there are multiple service rows with ‘serviceType’ MatrixStore, in which case only the one most recently updated should be in use, this script will update the UID on the service with the most recent lastModifiedDate and delete the other(s).

- remove occurrences of catdv.matrixstore_archive.service_mode from the server and service properties, via their respective control panels.

Changes

- significant update to plugin initialisation to prevent it blocking the server thread on start-up and simplify switching between in-server or standalone service for job processing. The same UID is now used for in-server and standalone services and the service mode is switched via a UI service config option, rather than system property. See Manage Service Command (Processing Tab – Service mode).

2.2.4 (2020-01-30)

Updates

- If running service standalone, update CatDV Service Control Panel to 1.3

Changes

- Fix standalone service to run with updated Service Control Panel

- Make feedback message more prominent for account operations on the manage service command

- On job queue, preserve job result fields on refresh, to prevent output field jumping every few seconds

- Fatally fail jobs where the associated clip is not found (deleted since job created) and include ‘Clip not found’ in status details.

- Fix to prevent job exceeding max retries

2.2.3 (2019-12-17)

- Add restore path mapping

- All service config duration values are now displayed, and may be entered, in h (hours), m (minutes), s (seconds) or any combination in that order, e.g. 2h, 2h 30m, 5m, 1m 30s, 1s.

- Fix plugin command for generating json metadata for a file on MatrixStore not to query on category for the time being

2.2.2betap1 (2019-11-06)

- Fix handling of “Server too busy to answer” error when calculating MD5 on MatrixStore for large files

- Fix for bug when job details/results fields remain populated when job table has emptied (due to refresh / cancel)

2.2.2beta (2019-10-24)

- Require server version 8.0.4 for fix to #190925-110544-112, issue with table row auto-selection on web

- Require server version 8.0.4 for fix to #190927-110649-832, cannot multi-select expandable rows on web

- Update documentation for Media Store Service Path Mapping

- Update rest API call for generating archive parameters for a file on MatrixStore, so that it can be fed with a file path as well as an object id. If a file path is provided, the plugin will convert it to an archive object key as it would at the point of archive, taking media store mappings into account, then use the object key (with prefix /) to search against the MXFS_PATH metadata on MatrixStore.

2.2.1beta (2019-09-27)

- Fix Job Queue so that an attempt to cancel a failed/complete job does not report as a job cancelled

- Fix likely cause of unexpected service account re-ordering in manage service command

- Fix occasional erroneous display of unlock warning for on update/delete for accounts which are not in use

- Add logging property to change log level for MatrixStore SDK

2.2.0beta (2019-09-24)

- Initial version