This plugin supports the transfer of files between CatDV and ActiveScale. In addition, it supports the restore of files that were automatically transferred to ActiveScale Cold Storage from ActiveScale via bucket lifecycle policies.
This plugin does NOT support the direct transfer of files between CatDV and ActiveScale Cold Storage.
Before you start
To use this ActiveScale archiving plugin you need:
CatDV Server 10.1.2 or later
CatDV Pegasus or Enterprise Client 14.1 or later
CatDV Plugin licence 'ARCAS' (Rest API licence with multiple sessions)
To trigger ActiveScale file transfers from the Worker you need:
ActiveScale Worker Plugin 2.4, included in this installation as ActiveScaleWorker.catdv
In addition, to run the archive service standalone (outside the server):
CatDV Service Control Panel 2.0.0 or later
New Installation
IMPORTANT: You need the access keys for an ActiveScale account in order to proceed with installation.
Copy the whole directory extracted from this zip to the plugins directory:
e.g. on Unix systems: /usr/local/catdvServer/plugins e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins
Set optional server properties for the plugin, if desired:
Open 'Server Config' from the control panel
Enter required properties into the CatDV Server tab, 'Other' field. See Plugin server properties.
Restart the CatDV server from the control panel
Open the CatDV client (or log the client back in to the server)
Configure one or more service accounts (see Managing Service Accounts). The first account will initially be the default account, used to verify that ActiveScale is available and used as a fallback when necessary. See Manage service command for further information on the details and settings for a service account:
In the client, run Tools->Manage ActiveScale Archive Service
On the accounts tab enter the service account details. Mandatory details are flagged with an asterisk. Note that once a service account has been used to archive a clip, it will no longer be possible to update its identifier.
Optionally modify the defaults on the settings tab.
Click 'Add'.
Configure service settings (See Manage service command)
Optionally configure archiving, processing and UI settings for the plugin via the corresponding tabs on Tools->Manage ActiveScale Archive Service.
Set up Media Store Service Path Mappings to automatically map drives/folders to archive locations. If you plan to always use a single service account and a single bucket, you may skip this step and use default location mapping. However, it is still recommended to use this approach if you don’t want the file ‘paths’ on ActiveScale to include the path of the storage volume.
If Running archive service standalone, then configure and start the archive service via the service control panel.
Verify the service setup: In the client, run Tools->Manage ActiveScale Archive Service again. The service status for the default account should be 'Running (online)'. The status may be 'Running (offline)' if ActiveScale is not currently accessible.
IF the Worker IS NOT being installed
Delete ActiveScaleWorker.catdv from the ActiveScale plugin directory (see first step)
IF the Worker IS being installed:
Move ActiveScaleWorker.catdv FROM the ActiveScale plugin directory (see first step) TO the worker plugins directory:
e.g. on a Mac: /Library/Application Support/Square Box/Extensions e.g. on Windows: %ProgramData%\Square Box\Extensions e.g. on Linux: /usr/local/catdvWorker/extensions
If the worker is running, re-connect to the server so that the archive plugin fields are available.
Verify that the ActiveScale worker actions are available. It may be necessary to quit and re-open the worker to pick up the new worker actions.
Upgrade Installation
Make a note of the current plugin version from this document (or see latest version in Release Notes)
If running the archive service standalone, use the service control panel to stop the service.
Stop the CatDV server from the control panel
Copy the whole directory extracted from this zip to the plugins directory:
e.g. on unix systems: /usr/local/catdvServer/plugins e.g. on Windows: C:\Program Files (x86)\Square Box\CatDV Server\plugins
Remove or move the directory / files for the prior plugin from the plugins directory.
Carry out any 'Upgrade' instructions for each plugin version listed in the Release Notes above the last installed version, working back from the oldest applicable version.
Start the CatDV server from the control panel.
Read through the 'Changes' for each plugin version listed in the Release Notes above the last installed version, and go to Tools->Manage ActiveScale Archive Service to verify that the details / settings for each account and the archiving / processing / UI settings for the plugin are correct for this installation. See Manage service command for more details.
If Running Archive Service Standalone, then update and start the archive service via the service control panel.
Open the CatDV client (or log the client back in to the server).
Verify the service setup: In the client, run Tools->Manage ActiveScale Archive Service again. The service status should be 'Running (online)'. The status may be 'Running (offline)' if ActiveScale is not currently accessible.
Update (or delete) the ActiveScale worker plugin by following the instructions for the worker plugin in a new installation. If there is an older ActiveScale worker plugin file in the extensions directory with a versioned name, delete it.
Plugin server properties
When running the archive service "InPlugin", the following server properties must be set:
catdv.activescale_archive.licence_code = <generated licence code for plugin>
When running the archive service standalone, the properties for the archive service must include:
catdv.activescale_archive.licence_code = <generated licence code for plugin> catdv.rest_api.host = <CatDV server hostname> catdv.rest_api.port = <CatDV API port> catdv.rest_api.user = <CatDV API user> catdv.rest_api.password = <CatDV API user password> (** exclude if blank, for dev)
Typical rest API values for a development machine would be (password property omitted for a blank password):
catdv.rest_api.host = localhost catdv.rest_api.port = 8080 catdv.rest_api.user = Administrator
In addition, the following optional property may be set to turn on debug logging on both the server and standalone service:
catdv.activescale_archive.debug = true
Running Archive Service Standalone
By default, the service that handles archive jobs runs inside the CatDV server. The following is required to run the plugin's archive service on a separate machine from the CatDV server:
CatDV Service Control Panel 1.3 or later
To configure and start the standalone archive service using the service control panel:
Open the Manage service command in the client / web plugin UI
Verify that at least one service account has been configured
In the Processing tab, set the ‘Service mode’ to ‘standalone’ and click ‘Save’
Re-open the Manage Service Command and verify that the service is ‘Stopped’. If not, restart the CatDV server.
Install the CatDV Service Control Panel
Open the CatDV Service Control Panel
Click 'Service Config' to open the service configuration
Fill in license details on the Licensing tab
Optionally, fill in details of a REST endpoint on the REST Service tab
Fill in server connection details on the Server Connection tab (connection can be checked on the control panel)
Enter required service properties on the Settings tab. See Plugin server properties above. (**Note the server connection details are duplicated here for now, pending a plugin update to remove them)
On the Installation tab, note the install location ('Server Install Path')
On the file system, navigate to the install location and copy the plugin files into the plugins sub-directory.
Click 'Start' to start the archive service.
To update the standalone archive service:
The service should have been stopped via the Service Control Panel 'Stop' prior to stopping the CatDV Server.
Open the CatDV Service Control Panel
Add / edit service properties on the Settings tab if applicable.
Go to the install location (see Installation tab) and replace the plugin files in the plugins sub-directory with those for the latest version of the plugin.
Click 'Start' to start the archive service.
ActiveScale Configuration
To use the plugin, at least one ActiveScale account is required which has access to the bucket(s) to which media will be archived. The values required to configure a service account in the ActiveScale plugin are:
endpoint - endpoint for an ActiveScale archive
access key - access key from credentials of ActiveScale user
secret key - secret key from credentials of ActiveScale user
default bucket - name of a bucket on ActiveScale which is accessible to the user with the above credentials and can therefore be used to verify the connection, also used as the fallback bucket the first time a user attempts to copy / move files to ActiveScale
Media Store Service Path Mappings
Media Stores should be used to automatically map file storage to archive locations by adding a service path to each media store. The service path determines the service account, bucket and (optional) folder path for the archived files from that file storage. Service accounts are set up via the Manage Service screen (see Managing Service Accounts) and are given an identifier when they are created.
For ActiveScale a service path has the format:
<serviceType>://<accountIdentifier>/<bucketName>
or
<serviceType>://<accountIdentifier>/<bucketName>/<folderPrefix>
For example, if a Media Store were set up with the following two entries:
/Volumes/dance-videos/drafts
activescale://squarebox-test/video-test/dance
Then a clip located at:
/Volumes/dance-videos/drafts/greatest-show/this-is-me
Would be archived:
using the credentials from the 'squarebox-test' account
into the bucket 'video-test'
with a path of '/dance/greatest-show/this-is-me'
Each Media Store is intended to represent a distinct segment of physical storage so, to avoid ambiguity, no path should be present in multiple media stores and paths should not overlap (e.g. there should not co-exist media stores for /Volumes/dance-videos/drafts and /Volumes/dance-videos/drafts/special).
To avoid any clash or overwrite of files with the same path and name from different drives, make sure they are mapped to either distinct buckets or distinct folders within the same bucket.
The 'folder path' can be set or overridden on archive if 'location' is set as an allowed override. See Manage service command (UI TAB).
Restore path mapping
If a restore location is not writable when a restore job is run, the plugin will use Media Stores to attempt to find an alternative writable restore location. If there is a Media Store with a path matching the restore location, the plugin will check whether any of the other paths in that Media Store are writable and substitute the root directory of the restore location accordingly.
The restore location can be overridden when a restore job is scheduled, if the option to override it is turned on. In earlier versions of the plugin, in this case no file hierarchy was preserved – all files were written directly to the specified directory. Now the plugin will use Media Stores to try and preserve file hierarchy for restored files. If any Media Store path is found that matches the default restore location, it will be used to substitute the root directory of the default restore location with the override.
Note that if many files are restored together to an overridden location, some files may have hierarchy preserved and some may not, depending on whether matching Media Store paths are found.
Using the plugin
The plugin comprises various commands that are available from the Tools menu in the client.
File transfers are initiated by the schedule backup / archive / restore commands but are carried out by a separate process. This means that the client does not automatically reflect updates to a clip on completion of a transfer. Use 'Server->Refresh Window' in the client to pick up any changes during / after transfer. The plugin includes the following commands:
Command | Description |
Manage ActiveScale Archive Service | View full status details of the archive service and manage the service account(s) required to access ActiveScale archives. Service account details include region / endpoint, access key, secret key, default bucket name and optional CMK ID. If a connection to ActiveScale cannot be made an error is displayed and the values are not saved. |
View ActiveScale Archive Service Job Queue | Lists jobs, which can be filtered by status (Pending, Complete, Failed, All). It provides the capability to view job contents, view job results and cancel waiting or queued jobs. IMPORTANT: The number of jobs displayed in any view is limited to the ‘Max no jobs to display’ configured on the UI tab of manage service. To see all jobs use the web admin job queue. |
View ActiveScale Jobs for Clip(s) | [*beta] Provides similar functionality to the job queue but displays all jobs / jobs of a given status for the selected clip(s) only, including clip members where applicable. For folder type clips (folder / playlist / version set / multicam) this incorporates all clips & members in the hierarchy. |
Schedule copy to ActiveScale | Adds a copy job for each clip selected (if multiple clips reference the same source media only one job is created). Copy jobs can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and ActiveScale is accessible). When the copy job is run, the source media associated with the clip is copied from the file system to ActiveScale and the original file is preserved. |
Schedule move to ActiveScale | As copy but on successful transfer of the file to storage the job attempts to delete the original file. If for some reason the deletion fails, this does not affect the job status (or archive status of the clip) but "Purge failed" is recorded in the job data. |
Schedule restore from ActiveScale | Adds a restore job for each clip selected (if multiple clips reference the same source media only one job is created). Restores can be scheduled when the archive service is not running or is offline but will only be run when the archive service is online (running and ActiveScale is accessible). When the restore job is run, the source media associated with the clip is copied from ActiveScale to the file system. |
Purge files archived to ActiveScale | Deletes the source media associated with the selected clips if they have been successfully archived. |
Recover ActiveScale archive data | Attempts to recover the archive data for the selected clip(s) so that they can subsequently be restored and/or purged. If the archive key is missing, the same approach as copy / move is used to determine the location of the file on ActiveScale, including media store mappings and overrides where applicable. The location is used to try and locate the file on ActiveScale. If a clip has the ‘Last archived parameters’ filled in then it will be skipped unless it has a non-complete archive status, in which case the last archived parameters will be re-instated. Otherwise: A combination of the source media level archive status (if present), last complete archive job (if present) and archive key are then used to recover as much of the archive data as possible. |
Managing Service Accounts
The accounts tab of Tools->Manage ActiveScale Archive Service provides facilities to manage the service account(s) used to connect to ActiveScale archives. A single account is usually sufficient to operate the plugin. However, if required, there is an add-on which enables multiple accounts to be set up for security purposes etc. This is enabled as an add-on via a special server config property.
When there are multiple accounts, the account used may be determined automatically via Media Store Service Path Mappings or selected by the user on archive / restore if allow override of Account ID is turned on from the UI tab. See Manage Service Command (UI Tab).
An account which is in use, i.e. has been used to archive files, must be unlocked before it can be updated or deleted. This is to ensure that accidental changes cannot be made to an account such that files can no longer be restored. Note it is not possible to change the account identifier on an in-use account.
The accounts tab provides the following operations:
Button | Description |
Clear | Clears the current selection / details so that only default values are filled in. |
Set Default | Updates the default account to the current selected account. |
Add | Creates a new account with the specified details. |
Unlock | Unlocks an in-use account in preparation for update / delete. |
Update | Updates the selected account with the specified details. If an account is in use, it must be unlocked before it can be updated. Please take care when updating an account which is in use, as changing some settings could break the restore of files archived with that account. For example, if the account key is switched to one which does not have the same access or if any encryption settings are changed. Note it is not possible to change the account identifier on an in-use account. |
Delete | Deletes the selected account. Please take care when deleting an account which is in use, as it will no longer be possible to restore any clips arched with that account. This feature is intended for removing test accounts and any applicable clips should be deleted or restored and re-archived. |
Manage service command
The following can be configured from Tools->Manage ActiveScale Archive Service in the web or desktop UI for the plugin:
ACCOUNTS TAB / DETAILS (for an account): | |
Field | Description |
Account Identifier | Identifying name for this service account. This may contain only alpha-numerics and hyphens. Note it is not possible to change the account identifier on an in-use account. |
Endpoint | Endpoint for a ActiveScale storage |
Access Key | Access key for connecting to this service account |
Secret Key | Secret key for connecting to this service account |
Default Bucket Name | The bucket that will be used as the archive fallback for this account by default. This is not applicable when a media store service mapping applies. Otherwise, if the UI tab settings allow the user to override the bucket then this will only be used as the default the first time the user does an archive. Subsequent archives by the same user will default to the last value they entered. |
Default Cold Storage Restore Priority | The restore priority that will be applied to restores from cold storage by default. If the UI tab settings allow the user to override this restore priority, then this will only be used as the default the first time a user does a restore. Subsequent restores by the same user will default to the last value they selected. |
ACCOUNTS TAB / SETTINGS (for an account): | |
Thread pool size | Maximum number of threads the TransferManager will use to simultaneously transfer (parts of) files to and from ActiveScale. Note, there is a separate TransferManager for each ActiveScale account set up in the plugin, so this should be taken into consideration. |
Multipart upload threshold (MB) | The file size at which the AWS SDK will start breaking files into multiple parts for upload. |
Minimum upload part size (MB) | The minimum size of each part for a multiple part upload. |
Cold storage restore expiration (days) | The expiration period, as a number of days, for which a file restored from cold storage will be accessible directly from ActiveScale (restore from cold storage is always temporary). The default is 2 days. |
Host Addressing | Determines whether the plugin uses virtual host addressing or resource path addressing to connect with the S3 service. Valid values are 'virtual' or 'pathStyle'. The default value is 'virtual'. |
ARCHIVING TAB: | |
Exclude archived files from archive | Skip files that have already been archived when scheduling jobs to copy / move to archive, unless they currently have a copy / move failed status. Default is “yes” for new installs. Note, if the file(s) were previously copied, an attempt to move the file will simply be skipped, it will not purge the file. When this option is turned on, use the purge operation to purge files which have already been copied. This option can be temporarily disabled if it is necessary to resubmit files for some reason. |
Exclude existing files from restore | Skip files that already exist when scheduling restore jobs. Default is “yes” for new installs. |
Purge directories | Determines whether or not empty directories are deleted when purging (or moving) files. |
PROCESSING TAB: | |
Service mode | NB Always ensure that the standalone service is NOT running prior to changing this value. Determines whether the job processing service runs in-server (as a thread in the CatDV Server) or standalone (via the Service Control Panel). The initial/default value is in-server. When this value is switched, the plugin will attempt to start or stop the in-server service as appropriate. If this does not appear to have been successful, as reported on the Overview tab, then it will be necessary to restart the CatDV server to apply the change. |
Concurrent job limit | Determines the number of transfers that the archive service will attempt to run concurrently. The default value is 4. |
Number Delayed retries | The number of times a waiting job is retried at an interval which increases exponentially. The default value is 10, which is equivalent to roughly 34 hours elapsed running time of the archive service. |
Maximum retry delay (time period) | Limits the maximum time period between retries of waiting jobs. The default value is blank. |
Loop delay (time period) | Determines the frequency with which the archive service checks the Job queue and attempts to run outstanding Jobs. Defaults to 30s. |
Retry running job delay (time period) | Determines the time period after which a Job which is running will be restarted if it has not been updated during that period. Defaults to 1h. |
UI TAB: | |
Restrict command access | Restricts the specified plugin commands to sys admin users. Can hide: · 'config' commands only (i.e. Manage Service) · ‘config / archive’ (i.e. to restrict copy and move to archive but not restore and purge) · ‘config / archive / purge’ (i.e. to restrict copy, move and purge but not restore) · ‘all (excl queue)’ · 'all' commands NB a server restart is required to ensure that this change is picked up under all circumstances. Default is ‘none’. |
Max no jobs to display | The maximum number of jobs which will be listed in the job queue. This overrides days_to_display. The default value is 1000. |
Allow overrides | Enables the facility for users to override one or more parameters at the point of archive or restore. |
**Any duration can be set in h (hours), m (minutes), s (seconds) or any combination in that order, e.g. 2h, 2h 30m, 5m, 1m 30s, 1s.
Archive details pane
The plugin automatically creates a panel entitled "ActiveScale Archive" containing the clip metadata which describes it's ActiveScale archive state, including:
Field | Description |
squarebox.catdv.archive.ActiveScale.serviceType | Type of service responsible for file transfer |
squarebox.catdv.archive.ActiveScale.serviceName | Name of service responsible for file transfer |
squarebox.catdv.archive.ActiveScale.status | Archive status |
squarebox.catdv.archive.ActiveScale.location | Location of file in storage |
squarebox.catdv.archive.ActiveScale.restoreLocation | Location file will be (if job pending) or was last restored to |
squarebox.catdv.archive.ActiveScale.date | Date (timestamp) of latest change in archive status |
squarebox.catdv.archive.ActiveScale.dateLastArchived | Date last archived |
squarebox.catdv.archive.ActiveScale.dateLastRestored | Date last restored |
squarebox.catdv.archive.ActiveScale.numArchives | The number of times the clip has been successfully archived |
squarebox.catdv.archive.ActiveScale.archiveKey | Identifier of file in storage |
squarebox.catdv.archive.ActiveScale.batchID | Identifies the batch of files with which the clip was archived |
squarebox.catdv.archive.ActiveScale.jobID | Identifier of current / latest archive job |
squarebox.catdv.archive.ActiveScale.parentJobID | n/a for ActiveScale (related to bulk archives) |
squarebox.catdv.archive.ActiveScale.userId | ID of user initiating current / latest transfer |
squarebox.catdv.archive.ActiveScale.historyJson | Record of all archive activity in json format |
squarebox.catdv.archive.ActiveScale.history | Record of all archive activity (including before historyJson added) |
squarebox.catdv.archive.ActiveScale.purged | Indicates whether or not file has been purged by plugin (reset on restore) |
squarebox.catdv.archive.ActiveScale.purgeError | Details of purge failure |
squarebox.catdv.archive.ActiveScale.accountIdentifier | Identifier of the ActiveScale Storage Account in CatDV |
squarebox.catdv.archive.ActiveScale.regionName | Endpoint of the ActiveScale archive |
squarebox.catdv.archive.ActiveScale.bucketName | Name of bucket to transfer file to / from |
Known Issues
Restore can overwrite read only files.
License Code
IMPORTANT: You may install and use this software only in accordance with the terms of the CatDV Server 8 license agreement.
Square Box Systems Ltd.
September 2023
Release notes
2.4p1 (2023-09-08)
- Fix for archiving files from a mapped media path - Fix to exclude service paths when looking for a writable path for restore
- Amend the mechanism to get the next job to process to try and prevent an infinite loop for clips with corrupt archive data. NB jobs for such clips will be consistently skipped and need to be cancelled. The only indication of this will be info entries in the logs (it isn’t necessary for plugin debug to be on to see these).
2.4 (2023-02-28)
Upgrade
- Check requirements from ‘Before you start’
- Check whether CatDV server DB has an index on jobID on the jobResult table. Add this index if it is missing.
- Upgrade worker archive plugin (included in this release)
Changes
- Add support for nested folder clips (folder / playlist / version-set / multicam)
- Add support for lightweight directory clips
- New (beta) command “View … Jobs for Clip(s)”, to aid monitoring folder / lightweight clips and for troubleshooting.
- Minimise duplicate failure entries in archive history
- Remove days to display limitation and UI config option for job queue. Limit by max jobs only.
- Remove stalled job delay / stalled indicator on running jobs (thread dump now only on running time out)
- Improve error reporting by checking for file not found before passing a file to Amazon’s TransferManager
- Fix for NumberFormatException parsing clip JSON from server containing proxyOffset
2.3.x (n/a)
These release numbers were used for beta releases of multiple data movers, which has been released at 3.x
2.2.14p1 (2022-11-18)
- Add back host addressing as an editable parameter
- When updating service accounts, display account unlocked even if account already unlocked
2.2.14 (2022-11-20)
- Enable purge of externally archived files which have no user metadata on S3
- Set last modified date on restore of externally archived files which have no user metadata on S3
- Dump threads when a job stalls or times out, for troubleshooting purposes
- Minimise duplicate failure entries in archive history
- Fix for out of bounds exception triggering recover archive data from the worker
- Modify thread syncing for job processing to improve performance and attempt to resolve a thread sync issue that intermittently causes jobs to be re-run after completion, resulting in an MD5 error for a move job.
2.2.13p1 (2022-04-08)
Upgrade
- Worker plugin to 2.2.13p1
Changes
- Add recover archive data plugin command (UI and worker)
- Add option ‘All (excl queue)’ to restrict command access options
- Hide location mapping option, which is not applicable to a new plugin
- Replace references to ‘glacier’ in plugin UI and doc with ‘cold storage’
- Replace references ‘restore tier’ in plugin UI and doc with ‘restore priority’
- Removed ‘bulk’ as a restore priority option for ActiveScale
2.2.13 (2022-03-21)
- Initial version of ActiveScale plugin (S3 plugin branded for ActiveScale)
Copyright © Square Box Systems Ltd. 2002-2023. All rights reserved.