1. Byosphere® Data Uploader Guide
1.1. Introduction
The Protein Metrics Byosphere® Data Uploader is a server-based system which continuously uploads new or newly edited files to the Byosphere server. For example, as MS files are generated or copied to a designated source folder, they will automatically upload to Byosphere. In addition, the Data Uploader compresses MS sample files and folders to the proprietary *.pacq format, rendering them readable in Byosphere Desktop analyses. The Data Uploader is included in the Byosphere system.
1.2. System Requirements
The minimum hardware/software requirements for the machine that will host the Data Uploader is listed below:
| CPU | Memory | Storage | OS | Network |
|---|---|---|---|---|
| >= 2 cores | >= 16 GB | >= 100 GB | Windows Server 2022 | >= 1 GB/s |
The machine will require access to the file system with the files/folders to be uploaded.
1.3. Install the Automation Server Package
Ideally, the Automation Server should be installed in an environment that is independent from the Analysis Service so that they do not compete for the same resources.
- Download the Automation Server installer to the local environment and run the installer.
- On the local environment, run Byosphere to initialize the server URL in the configuration file and test the connection:
Run
\ProteinMetrics\PMI-Byosphere-Client\Base\PMi-Byosphere-Client.exeChoose Server > Configure, enter the Byosphere server URL, and click OK
1.4. Configure the Automation Server
Modify appsettings.json in the installed PMIAutomationService directory as follows:
| Parameter, Req’d & Default value | Note |
|---|---|
|
Parameter: Host Required: Yes |
Byosphere Application Server IP address |
|
Parameter: UseAPIToken Required: Yes, if using API token |
If true, set Windows Credential to use API token set on Application Server (see below). An API token is recommended for Byosphere v 4.1 and above. |
|
Parameter: KeychainNameSpace Default: PmiAutomationCredentials |
Must be consistent with the credentials set up using see notes below |
| Parameter, Req’d & Default value | Note |
|---|---|
|
Parameter: LocalWorkingPath Default: |
Folder location to store temporary files generated and then deleted during uploads. The default value will be used if the customer value is not specified or accessible. |
|
Parameter: ExecPath Default: |
The console.exe executable path from the Protein Metrics Byosphere Client software installation |
|
Parameter: BaseDir Required: On prem only |
This is a required parameter for OnPrem only. This is the NAS mount drive. This setting in Windows must match its counterpart in App Server’s base_dir in Linux. Example:
|
|
Parameter: AutomationConfigurationID Required: Yes |
The value is the Automation configuration ID in the Admin Portal. This is a required parameter for Data Uploader and Auto Processor. It is not required for Watcher. |
| Parameter, Req’d & Default value | Note |
|---|---|
| Parameter: LastScanTime |
This value is auto-created / updated by the last program run. Format is "yyyy-MM-dd HH:mm:ss" Example: |
|
Parameter: ConfirmFolderCanPacq Default: false |
If true, checks each folder before upload to confirm if it is an MS folder. This is used if MS folders contain file extensions that are also listed in the SearchPattern parameter, (e.g. .csv, .txt, .xml). Under those circumstances, this setting avoids a file upload of the SearchPattern extension in a separate folder. Note: this parameter should not be set if unneeded since it will significantly affect performance. |
| Parameter: RollingWindowDelta |
The unit is second. The Automation will check for new data using time stamp setup in Data will be uploaded using the following, given a 3600 second or 1 hour delta effectively checking the file against 12-20-22 11:30 AM.
If no value is specified, no delta window will be applied. Example: |
| Parameter: CustomSettings |
CustomSettings is an array of custom codes. Custom code will be executed in the order they are listed. Parameter includes
MetadataPath can contain output from the custom code. Contact support@proteinmetrics.com for more information on integration with third party tools through custom codes. Examples,
|
| Parameter: ApplyMetadataToExistingFile |
When
|
| Parameter: SkipPacqHashCheck | Automation relies on pacq hashes to detect modifications and trigger re-uploads. However, for immutable MS files, this check can be bypassed. By default (false), pacq hash comparison occurs. Setting this value to true skips the pacq hash check. |
| Parameter: SkipFileHashCheck | Automation relies on file hashes to detect modifications and trigger re-uploads. However, for immutable non-MS files, this check can be bypassed. By default (false), file hash comparison occurs. Setting this value to true skips the file hash check. |
| Parameter: LocalInputRoot (optional) |
Automation by default honors local folder hierarchy and will create a matching folder hierarchy in Byosphere. LocalInputRoot parameter allows user to define common path that will be skipped during Byosphere subfolder creation processor therefore redefine a root folder. LocalInputRoot is case insensitive. LocalInputPath and LocalInputRoot are expected to have the same path format of Windows, Linux, or UNC path. Examples in Linux or Windows filepath format: LocalInputPath c:/a/b/z, LocalInputRoot c:/a/b => common path c:/a/b -> data will be uploaded to Output Folder > z LocalInputPath c:/a/b, LocalInputRoot empty => no common path -> data will be uploaded to Output Folder > a > b LocalInputPath c:ab, LocalInputRoot c:m => no common path -> data will be uploaded to Output Folder > a > b LocalInputPath c:abz, LocalInputRoot c:abzy => common path c:/a/b/z -> data will be uploaded to Output Folder |
| Parameter: UserGroupIds (optional) |
Specifies the ids of user groups that should be assigned to the new folders created when uploading a file. The groups will be assigned to all uploaded folders or to subfolders from the level specified by UserGroupSubfolderLevel. Example: "UserGroupIds": "2,3" |
|
Parameter: UserGroupSubfolderLevel (optional) Default: 1 |
Specifies the level of LocalInputPath directory from where UserGroupIds should be applied when uploading folders. When this parameter is not provided or its value is less than or equal to 1, UserGroupIds will be applied to all new folders. Examples: "LocalInputPath": "C:/a/b/c/d/e", "UserGroupIds": "2,3" Example 1: "UserGroupSubfolderLevel ": 1 Result: User groups will be applied to all subfolders: a,b,c,d,e Example 2: "UserGroupSubfolderLevel ": 3 Result: User groups will be applied to the following subfolders: c,d,e Example 3 (when LocalInputRoot is provided): "LocalInputRoot": "C:/a/b/c", "UserGroupSubfolderLevel ": 3 Result: User groups will be applied to the following subfolders: d,e (subfolder 'c' will not be uploaded due to LocalInputRoot value) |
| Parameter: CheckCreationTime |
Default to false. When set to false, current behavior will be observed where only LastWriteTime is checked. This is the common case where data comes off acquisition machine, Modified is later than Created. When set to true, we will check both CreationTime and LastWriteTime, and use the later of the two. This is the case when a file is copied from one drive to another drive, Created is later than Modified. |
Note: The Data Uploader supports the upload of Chromeleon files for version. 7.2 and above. Contact support@proteinmetrics.com for more information and for the PMI Chromeleon Watcher Guide.pdf
1.4.1. Find Byosphere Upload Folder ID
Byosphere folder IDs are displayed when uploading to or downloading from a selected folder. To identify the folder ID, do the following:
- Create the upload folder in Byosphere. If the Keychain credentials for the Data Uploader is not a Super User, then add a user group for that user name with File Editor, Folder Editor and Viewer privileges.
- In Byosphere Desktop, login to the server and choose Server > Upload.
- In the Server file/folder cell, click the “…” button.
- Select the upload destination folder at left and click Choose.
- The folder ID is displayed in the Upload file/folder dialog, in this example “2026”:
Figure 1.1 Identify the Byosphere upload folder ID
1.4.2. Run as Windows Service
Protein Metrics Automation Server is recommended to run as a Windows Service rather than as a console application. The executable sc.exe in CMD can also be used to create, start, stop, or delete a Windows Service:
:: Create a Windows Service for the Automation Server
sc create PmiAutomationService DisplayName="PMI Automation Server" binPath="C:\Program Files\ProteinMetrics\Pmi-Automation-Service\PmiAutomationService.exe"
:: Start a Windows Service
sc start PmiAutomationService
:: Stop a Windows Service
sc stop PmiAutomationService
:: Delete a Windows Service
sc delete PmiAutomationService
After configuring and verifying the setup, it is recommended to set the Protein Metrics Automation Server Start Type to Auto. Windows services can be managed in the Windows Services App:
Figure 1.2 Window Services App
The Log On account for Protein Metrics Automation Service can be a Local Service:
Figure 1.3 Protein Metrics Automation Service Local logon account
Alternatively, the Log On account for Protein Metrics Automation Service can be an individual user:
Figure 1.4 Protein Metrics Automation Service Administrator logon account
Note: After verifying the setup, Automation Server failure recovery can be adjusted according to Company IT policies.
Figure 1.5 Automation Server failure recovery
1.4.3. Windows Credential for Automation Server
Automation Server uses Windows Credential Store to avoid exposing user passwords in the configuration. Commands are run in the CMD console. For Administrator log on services, a simple CMD window is used. For Local System log on services, there are many tools to access Local System account. One way is to use https://docs.microsoft.com/en-us/sysinternals/downloads/pstools
- Start CMD, Run as administrator
- Execute the following command:
psexec -i -s cmd.exe
- This will launch a new CMD window granting Local System account access
- Windows Credentials setup can be executed from the Local System account
Figure 1.6 Accessing Local System account
To configure the API token used in App Server authentication for Automation Server:
- Follow App Server Guidance on setup API Token
- Open a CMD window as shown above.
- Customize and run the following commands:
PMi-Keychain-Manager --delete --namespace=PmiAutomationCredentials
PMi-Keychain-Manager --add --namespace=PmiAutomationCredentials --key=server --value="https://byosphere.company.com"
PMi-Keychain-Manager --add --namespace=PmiAutomationCredentials --key=api-token --value=ABCDEabcde
PMi-Keychain-Manager --read --namespace=PmiAutomationCredentials --key=api-token --context-data="server=https://byosphere.company.com"
The final line will display the API Token value for the Byosphere server URL. This user account assigned to the API Token requires the Super User privilege and the Contributor entitlement.
Note: Credentials only need to be setup one. In the future updates, run the last line to verify.
Note: Automation Server is set to log on as Local System by default. Depending on Company IT policies, Windows Credential might be necessarily to setup from Local System account. To find out more, please refer to https://docs.microsoft.com/en-us/windows/win32/services/localsystem-account and
https://docs.microsoft.com/en-us/windows/win32/ad/the-localsystem-accountExample of a successful Automation Server startup using username and password: expect to see no errors:
Figure 1.7 Example of Successful Automation Server startup
Figure 1.8 Example of Failed Automation Server startup
1.5. Application Notes
- New files are identified by content, not by name. If a file is edited, it will be uploaded as new version over the existing file with that name.
- Files which have MS sample extensions but are not recognized by the Byos application as MS files or folders (for example, corrupted data) are uploaded but not compressed into *.pacq format. Multiple versions of these files may then be generated.
- Log files written to the
PMIAutomationService\Logsdirectory can be renamed or configured by editing the filePMIAutomationService\NLog.config. For more information, please refer to https://github.com/nlog/nlog/wiki/Configuration-file or contact support@proteinmetrics.com. Note that log level can be Debug, Info, Warning, Error, and Fatal. It is recommended to start with Debug level logging during initial deployment to enable Protein Metrics to better troubleshoot any issues. It is recommended to use Info in normal operation; Warning or high to minimize logging. User can delete log files atPMIAutomationService\Logssubdirectory if log information is no longer needed.
Branch: release/2025.09
Compile Date: 2025-Oct-03
Compile Time: 12:00:53