1. Byosphere® Automated Processor Guide
1.1. Introduction
The Protein Metrics Byosphere® Automated Processor is a server-based product which automatically generates analyses from MS files as they are written to source directories. When MS files are copied to designated source folders, workflows are generated in a designated directory and the MS files used in them are automatically uploaded to the Byosphere server and compressed to the proprietary *.pacq format. New workflows are then generated which use the server sample references and these workflows are then submitted to the server.
The product contains two parts:
- Watchers identify newly added or updated MS files and generates workflows based on provided templates, FASTA files and specialized instructions. Different Watchers can be used to create different kinds of workflows based on the kinds of MS files and their source directories.
- Automation Service, when in AutoProcess mode, identifies workflows created by the Watchers, uploads and compresses the MS files and submits equivalent workflows as analyses, using the server versions of the MS files. The same Automation Service, when in AutoUpload mode, serves as the Data Uploader, which is available with all Byosphere systems.
For more information about the Data Uploader, see Byosphere 05 Data Uploader Guide.pdf.
1.2. System Requirements
The minimum hardware/software requirements for the machine that will host the Data Uploader is listed below:
| CPU | Memory | Storage | OS | Network |
|---|---|---|---|---|
| >= 2 cores | >= 16 GB | >= 100 GB | Windows Server 2022 | >= 1 GB/s |
The machine will require access to the file system with the files/folders to be uploaded.
1.3. Install the Automation Server Package
Ideally, the Automation Server should be installed in an environment that is independent from the Analysis Service so that they do not compete for the same resources.
- Download the Automation Server installer to the local environment and run the installer.
- On the local environment, run Byosphere to initialize the server URL in the configuration file and test the connection:
Run
\ProteinMetrics\PMI-Byosphere-Client\Base\PMi-Byosphere-Client.exeChoose Server > Configure, enter the Byosphere server URL, and click OK
1.4. Configure the Automation Server
Modify appsettings.json in the installed PMIAutomationService directory as follows:
| Parameter, Req’d & Default value | Note |
|---|---|
|
Parameter: Host Required: Yes |
Byosphere Application Server IP address |
|
Parameter: UseAPIToken Required: Yes, if using API Token |
If true, set Windows Credential to use API token set on Application Server (see below). An API token is recommended for Byosphere v 4.1 and above. |
|
Parameter: KeychainNameSpace Default: PmiAutomationCredentials |
Must be consistent with the credentials set up using see notes below |
|
Parameter: LocalWorkingPath Default: |
Folder location to store temporary files generated and then deleted during uploads. The default value will be used if the customer value is not specified or accessible. |
|
Parameter: ExecPath Default: |
The console.exe executable path from the Protein Metrics Byosphere Client software installation |
|
Parameter: BaseDir Required: On prem only |
This is a required parameter for OnPrem only. This is the NAS mount drive. This setting in Windows must match its counterpart in App Server’s base_dir in Linux. Example:
|
| Parameter: AutomationConfigurationID | The value is the Automation configuration ID in the Admin Portal. This is a required parameter for Data Uploader and Auto Processor. It is not required for Watcher. |
| Parameter, Req’d & Default value | Note |
|---|---|
|
Parameter: ProjectUploadFolderID Required: Yes Default: 0 |
ID of the Byosphere server destination folder for submitted analyses using the Automated Processor; The user specified by the API token must be a Super User to create a root-level folder (the default value ID=0), or have File Editor, Folder Editor and Viewer privileges for a specified Byosphere Web Client folder. For the Automated Processor, project files created by the analysis are uploaded to this folder tree. Output Folder and ProjectUploadFolderID can be the same folder. To find a folder’s ID, see the Find Byosphere Upload Folder ID section below. |
| Parameter: AutomatedMode |
This parameter is required for Watcher. To run the executable as a Watcher, set it as:
In this mode, samples, workflow templates and other inputs are used to generate workflows in the directory defined by the Output parameter (see the Configure the Watcher section below). |
| Parameter, Req’d & Default value | Note |
|---|---|
|
Parameter: SearchPattern Required: Yes |
Extensions of all files (including MS files) to be uploaded. all other file types will be ignored. If AutoProcess is true, only MS files will be uploaded. |
| Parameter: Comment | Optional text to save in the Comment field of the uploaded file version |
|
Parameter: DeleteAfterUpload Default: false |
If true, input file or directory will be deleted upon successful upload to save space. Only SearchPattern/MSFile matched files/folders are deleted. |
|
Parameter: WriteLastScan Default: false |
If true, last scan time will be auto-created / updated by the program and written to the configuration file. Note: when set to true, the LastScanTime line should be removed (along with the comma after “ |
|
Parameter: LastScanTime Required: No |
This this value is auto-created / updated by the last program run. If both Modified After and LastScanTime exist, later of the two values will be in use. Format is "yyyy-MM-dd HH:mm:ss" Example: |
| Parameter: RollingWindowDelta |
The unit is second. The Automation will check for new data using time stamp setup in Data will be uploaded using the following, given a 3600 second or 1 hour delta effectively checking the file against 12-20-22 11:30 AM.
If no value is specified, no delta window will be applied. Example: |
| Parameter: ApplyMetadataToExistingFile |
When
|
| Parameter: CustomSettings |
CustomSettings is an array of custom codes. Custom code will be executed in the order they are listed. Parameter includes
MetadataPath can contain output from the custom code. Contact support@proteinmetrics.com for more information on integration with third party tools through custom codes. Examples,
|
| Parameter: ApplySampleMetadataToProject |
False When True, the metadata for the first sample added to a workflow is automatically included in the project. |
| Parameter: LocalInputRoot (optional) |
Automation by default honors local folder hierarchy and will create a matching folder hierarchy in Byosphere. LocalInputRoot parameter allows user to define common path that will be skipped during Byosphere subfolder creation processor therefore redefine a root folder. LocalInputRoot is case insensitive. LocalInputPath and LocalInputRoot are expected to have the same path format of Windows, Linux, or UNC path. Examples in Linux or Windows filepath format: LocalInputPath c:/a/b/z, LocalInputRoot c:/a/b => common path c:/a/b -> data will be uploaded to Output Folder > z LocalInputPath c:/a/b, LocalInputRoot empty => no common path -> data will be uploaded to Output Folder > a > b LocalInputPath c:ab, LocalInputRoot c:m => no common path -> data will be uploaded to Output Folder > a > b LocalInputPath c:abz, LocalInputRoot c:abzy => common path c:/a/b/z -> data will be uploaded to Output Folder |
| Parameter: UserGroupIds (optional) |
Specifies the ids of user groups that should be assigned to the new folders created when uploading a file. The groups will be assigned to all uploaded folders or to subfolders from the level specified by UserGroupSubfolderLevel. Example: "UserGroupIds": "2,3" |
|
Parameter: UserGroupSubfolderLevel (optional) Default: 1 |
Specifies the level of LocalInputPath directory from where UserGroupIds should be applied when uploading folders. When this parameter is not provided or its value is less than or equal to 1, UserGroupIds will be applied to all new folders. Examples: "LocalInputPath": "C:/a/b/c/d/e", "UserGroupIds": "2,3" Example 1: "UserGroupSubfolderLevel ": 1 Result: User groups will be applied to all subfolders: a,b,c,d,e Example 2: "UserGroupSubfolderLevel ": 3 Result: User groups will be applied to the following subfolders: c,d,e Example 3 (when LocalInputRoot is provided): "LocalInputRoot": "C:/a/b/c", "UserGroupSubfolderLevel ": 3 Result: User groups will be applied to the following subfolders: d,e (subfolder 'c' will not be uploaded due to LocalInputRoot value) |
Note: The Automation Server supports metadata. The configuration can be customized so that the folder and file names are parsed into strings which can be saved with the file to specific metadata fields. For information about metadata customization, contact support@proteinmetrics.com.
1.5. Configure the Watcher
The installed PMIAutomationService folder is also used by the Watchers. After configuring this folder for the Automation Service, make a copy of the folder for each Watcher and name them accordingly. Next, configure the appsettings.json in each of the Watcher directories. Most of the Watcher *.json parameters that are shared with the Automation Server *.json parameters above will be assigned the same values. There are also specialized parameters for the Watchers to configure as follows:
| Parameter, Req’d & Default value | Note |
|---|---|
|
Parameter: LocalInputPath Required: Yes |
The path of the folder or root folder of subfolders containing the MS files to be uploaded and processed as analyses. |
|
Parameter: SubFolderLevel Default: 1 |
The subfolder level of the LocalInputPath directory where the MS files are found. For a value of 1, the MS files are found in the LocalInputPath directory; for a value of 2, the MS files are in folders under that directory, etc. |
|
Parameter: ProcessEntireFolder Default: false |
If true, all the MS files in folder at the SubFolderLevel as well as its subfolders are loaded into a single workflow Note: This must be set to false for |
|
Parameter: WflwOutputDir Required: Yes |
The directory where workflows are generated when BuildWflw is set to true, typically the LocalInputPath directory specified for the Automation Service |
|
Parameter: WflwTemplate Required: Yes |
The path or file name of the workflow file containing the workflow parameters, apart from samples, which are loaded from the LocalInputPath tree, and sequences, which are loaded from the Sequence path and fasta file. |
|
Parameter: Sequence Required: Yes |
The path or file name of the fasta file(s) containing any sequences to load. |
|
Parameter: ReportConfig Required: No |
The full path and file name of the report configuration Note: Local report configuration files can be added directly to the workflow templates. This is recommended if the Watcher applies to more than one workflow with different report types. In that use case, use absolute paths since relative paths are not supported. |
|
Parameter: SampleProteinCSV Required: No |
The full path and file name of the sample-protein *.csv file to be loaded into the workflow. |
|
Parameter: CustomJS Required: Yes |
The path and file name of a Custom.js file containing instructions for naming and parsing files and workflows. For help with customizing generated workflows, contact support@proteinmetrics.com
|
|
Parameter: AutomationTrigger Required: No |
When ProcessEntireFolder is true, a more precise way to trigger the Watcher is to use an automation trigger file. A sub-folder will be processed if and only if the filename designated as the AutomationTrigger is present in the folder. Example: "AutomationTrigger": "WatersmabEnd.raw" |
|
Parameter: ContinueAfterCustomCode Required: No Default: true |
If true, we will continue with BuildWflw after custom code is executed. If false, no action will be taken after the custom code is executed. |
1.5.1. Find Byosphere Upload Folder ID
Byosphere folder IDs are displayed when uploading to or downloading from a selected folder. To identify the folder ID, do the following:
- Create the upload folder in Byosphere. If the Keychain credentials for the Data Uploader is not a Super User, then add a user group for that user name with File Editor, Folder Editor and Viewer privileges.
- In Byosphere Desktop, login to the server and choose File > Upload.
- In the Server file/folder cell, click the
“…”button. - Select the upload destination folder at left and click Choose.
- The folder ID is displayed in the Upload file/folder dialog, in this example “2026”:
Figure 1.1 Identify the Byosphere upload folder ID
1.5.2. Run as Windows Service
The Protein Metrics Automation Server and the Watchers are recommended to run as Windows Services rather than as console applications. The executable sc.exe in CMD can also be used to create, start, stop, or delete a Windows Service:
:: Create a Windows Service for the Automation Server
sc create PmiAutomationService DisplayName="PMI Automation Server" binPath="C:\Program Files\ProteinMetrics\Pmi-Automation-Service\PmiAutomationService.exe"
:: Start a Windows Service
sc start PmiAutomationService
:: Stop a Windows Service
sc stop PmiAutomationService
:: Delete a Windows Service
sc delete PmiAutomationService
Repeat these steps for each Watcher Server. After configuring and verifying the setup, it is recommended to set the Protein Metrics Automation Server Start Type to Auto. Windows services can be managed in the Windows Services App:
Figure 1.2 Window Services App
The Log On account for Protein Metrics Automation Service can be a Local Service:
Figure 1.3 Protein Metrics Automation Service Local logon account
Alternatively, the Log On account for Protein Metrics Automation Service can be an individual user:
Figure 1.4 Protein Metrics Automation Service Administrator logon account
Note: After verifying the setup, Automation Server failure recovery can be adjusted according to Company IT policies.
Figure 1.5 Automation Server failure recovery
1.5.3. Windows Credential for Automation Server
Automation Server uses Windows Credential Store to avoid exposing user passwords in the configuration. Commands are run in the CMD console. For Administrator log on services, a simple CMD window is used. For Local System log on services, there are many tools to access Local System account. One way is to use https://docs.microsoft.com/en-us/sysinternals/downloads/pstools
- Start CMD, Run as administrator
- Execute the following command:
psexec -i -s cmd.exe
- This will launch a new CMD window granting Local System account access
- Windows Credentials setup can be executed from the Local System account
Figure 1.6 Accessing Local System account
To configure the password used in App Server authentication for Automation Server:
- Follow App Server Guidance on setup API Token
- Open a CMD window as shown above.
- Customize and run the following commands:
PMi-Keychain-Manager --delete --namespace=PmiAutomationCredentials
PMi-Keychain-Manager --add --namespace=PmiAutomationCredentials --key=server --value="https://byosphere.company.com"
PMi-Keychain-Manager --add --namespace=PmiAutomationCredentials --key=api-token --value=ABCDEabcde
PMi-Keychain-Manager --read --namespace=PmiAutomationCredentials --key=api-token --context-data="server=https://byosphere.company.com"
The final line will display the API Token value for the Byosphere server URL. This user account assigned to the API Token requires the Super User privilege and the Contributor entitlement.
Note: Credentials only need to be setup one. In the future updates, run the last line to verify.
Note: Automation Server is set to log on as Local System by default. Depending on Company IT policies, Windows Credential might be necessarily to setup from Local System account. To find out more, please refer to https://docs.microsoft.com/en-us/windows/win32/services/localsystem-account and
https://docs.microsoft.com/en-us/windows/win32/ad/the-localsystem-account
Example of a successful Automation Server startup using username and password: expect to see no errors:
Figure 1.7 Example of Successful Automation Server startup
Figure 1.8 Example of Failed Automation Server startup
1.6. Application Notes
- New files are identified by content, not by name. If a file is edited, it will be uploaded as new version over the existing file with that name.
- Files which have MS sample extensions but are not recognized by the Byos application as MS files or folders (for example, corrupted data) are uploaded but not compressed into *.pacq format. Multiple versions of these files may then be generated.
- Log files written to the
PMIAutomationService\Logsdirectory can be renamed or configured by editing the filePMIAutomationService\NLog.config. For more information, please refer to https://github.com/nlog/nlog/wiki/Configuration-file or contact Protein Metrics support. Note that log level can be Debug, Info, Warning, Error, and Fatal. It is recommended to start with Debug level logging during initial deployment to enable Protein Metrics to better troubleshoot any issues. It is recommended to use Info in normal operation; Warning or high to minimize logging. User can delete log files atPMIAutomationService\Logssubdirectory if log information is no longer needed.
Branch: release/2025.09
Compile Date: 2025-Oct-03
Compile Time: 12:00:53