Business Continuity with Enterprise Chromatography Data Systems
How do I operate my chromatography if the chromatography data system infrastructure fails?
Written by Robin Frey, Piotr Kuchta and Lukas Graubner
Introduction
Today, analytical laboratory methods such as chromatography are controlled via CDS within a wide range of industries and are subject to strict guidelines and regulations. Maintaining analytical activities in exceptional circumstances while meeting all legal requirements is essential for most companies.
Problem definition
These so-called chromatography data systems (CDS) are operated by companies, especially in the pharmaceutical industry, at enterprise level, through which adequate validation, corresponding change management and the associated documentation can be ensured. These systems usually consist of a centralized, or even cloud-based, infrastructure and dedicated clients which are localized next to the instruments for data recording (client-server infrastructure).
If there is a communication interruption (network connection interruption) between the client and server in such an infrastructure, the continuation of analytical activities must be guaranteed (business continuity). On the one hand, ongoing analyses must be fully completed (no interruption of data recording) and evaluated. On the other hand, it must be possible to create, start and evaluate new analyses accordingly.
Requirements
From a regulatory perspective, the data integrity principles according to ALCOA+ must be respected and complete traceability of the data must be guaranteed even in such an exceptional situation.
In addition, interruptions or outages in data recording and data processing must be prevented so that production processes remain largely undisturbed.
Realistic example scenario
During renovation work in the laboratory building, a major incident occurs and, among other things, the primary and secondary network cabling in the building's vertical cable shaft is damaged. As a result, the chromatography laboratory no longer has a network connection to the company network. As the server infrastructure of the chromatography data system is not located in the same building, access to the central data storage is also no longer possible. Repairing the cabling in the vertical cable shaft will certainly take three to five working days. However, the production rooms are not affected. The production of several products worth several million euros is planned for the outage period. If the release in the chromatography laboratory cannot take place, the manufactured goods cannot be released and production comes to a standstill with immense economic damage.
General description of the solutions
The three most widely used chromatography data systems all work with local client computers to which the chromatography instruments are directly connected. Depending on the manufacturer, these clients are called differently - Waters: LAC/E; Agilent: Agilent Instrument Controller AIC; Thermo Scientific: Instrument Controller IC or IPC. These local computers have the primary task of buffering the acquired data before it is finally stored on the server. In the event of a network failure, this buffering is the key to the continuation of ongoing chromatography. The local clients can also act as software clients for the chromatography data systems if a screen, keyboard and mouse are connected to them.
Questions to be answered
This leads to the following questions that need to be answered in the exceptional case of network interruption:
- Are software licenses and user administration still available if the network fails?
- Will ongoing measurement series continue to be measured without interruption?
- Can measurements be appended to an ongoing measurement series that was started before the interruption?
- Can the data of an ongoing measurement series that was started before the interruption also be evaluated during the interruption?
- Can a new measurement series be started?
- Can existing (validated) methods and report templates still be used?
- Can new measurement series be evaluated and released during an interruption?
- Is data measured during the interruption fully synchronized back to the server?Are all actions performed during the network outage fully traceable without manual documentation?
Available functions in the current versions of the three most widely used chromatography data systems
(in alphabetical order by manufacturer)
In the following three sections, the respective solution approach of the three selected manufacturers is described in detail. The next section will summarize and analyze how the above questions can be answered for the respective systems.
Agilent OpenLab CDS 2.8
Agilent calls the solution for bridging network failures between the server infrastructure and the AIC the Acquisition Failover Mode. The failover mode is triggered as soon as the connection between the AIC and the server infrastructure is interrupted. Any connection of a remote client is also informed about the loss of the connection and that it is possible to use failover mode.
To operate a device in failover mode, you must identify the corresponding AIC that belongs to the device and log on locally to the operating system of this computer, either locally with screen, keyboard and mouse, or if parts of the network are still working, via a remote desktop connection (RDP).
Open the OpenLab Control Panel on the AIC, which then suggests the failover mode option:
If you now use failover mode, all actions within the software are executed in the context of the OpenLab system user instead of the authenticated user. A corresponding note can be seen in the footer of the OpenLab client.
All methods, sequences and report templates that were used on an AIC in normal operation are saved in the cache and can be used in failover mode to create new individual injections or sequences. New methods can also be created and used. However, newly created methods are not automatically synchronized to the server when the connection to the server infrastructure is re-established.
Projects and project groups are synchronized to the AIC cache every 30 minutes during normal operation. All synchronized projects can also be used in failover mode.
In failover mode, only the device that is directly connected to the AIC can be accessed.
It is not possible to create new projects, configure new devices or reconfigure existing devices in failover mode. System settings cannot be changed either.
Data recording and evaluation is basically identical to normal operation, but with full system rights rather than the normally assigned user and access rights. Due to the failover mode, however, the following restrictions apply:
- Injections and sequences that were started or measured during normal operation cannot be edited or viewed in failover mode
- Snapshots (intermediate results) of sequences that were started during normal operation cannot be created
- Activity log entries that were generated during failover mode are synchronized back to the server after the connection is re-established and marked accordingly with failover mode. However, the corresponding name of the AIC is entered as the username, as no user administration is available in failover mode.
- Sequences and results that were created during failover mode are marked accordingly.
Data (result files) generated during failover mode remain in the cache on the AIC. Once the connection to the server has been restored, the user must manually synchronize them to the server using the “Failover Results Uploader” tool. Once the data has been synchronized to the server, it is removed from the local cache.
Every user who can log on to an AIC also has the option of using failover mode. All actions performed in failover mode are assigned to the system user, as no user authentication is required to start failover mode.
Even if new methods are created during failover mode, these are assigned to the system user in the audit trail.
As soon as the network connection is re-established after a failover mode, all sequences and samples recorded during the failover mode are marked accordingly in the activity logs and audit trails. In the instrument activity log, the system user is also replaced by the name of the AIC.
Agilent Technologies repeatedly points out that separate procedures must be developed on the customer side for the exceptional case in order to document the failover mode accordingly.
Thermo Scientific Chromeleon 7.3.2 MUd
Thermo Fisher calls its solution for ensuring business continuity during the scenario described Network Failure Protection (NFP). This is a standard feature of Chromeleon and does not require additional licensing. All system information is constantly synchronized on each local PC. The license is kept in the cache for 7 days. The required parts of the user administration are also synchronized and cached locally on the IPC.
Each measurement series started in Chromeleon, called a sequence in Chromeleon, is completely moved/copied to a specific data vault on the IPC, the so-called XVault. During the measurement and constantly synchronized with the server.
If the network fails, a complete copy of the data is available in the XVault on the IPC. The sequence continues without interruption.
After some time, the DataVault server is no longer accessible from the IPC and the client displays a yellow warning that the license will expire in 6 days. This is because the central license is cached for 7 days on each computer of the Chromeleon Enterprise system.
The data stored in the XVault can be accessed in a Chromeleon client session on the IPC indirectly via the device's queue. Chromeleon constantly attempts to synchronize the data with the server (Uploading).
After the sequence is completed, it is canceled after a timeout of the upload attempts and marked as “Upload Failed”.
In this state, it is possible to open the sequence and edit it without restriction, according to the privileges that the logged-in user has. If the report template is available as a copy in the sequence, an electronic report can even be created and electronically signed. If the report is linked to the sequence and is therefore on the server, no electronic report can be created unless another local report is used.
It is also possible to attach further samples to the existing sequence and measure them, but it is also possible to create new sequences (also as a copy of the sequences in the queue, provided that all objects are available locally). However, these must be assigned to the corresponding connected instrument during creation:
A message then appears stating that this sequence is only available from the current IPC.
Unfortunately, this message is not correct in Chromeleon 7.3.2 because the sequence is not automatically added to the queue. The sequence must be added to the queue manually.
The sequences created in this way are stored in the XVault and remain in the queue of the instrument, this is clearly shown in the name of the sequence path.
These sequences can be opened and edited from the queue of the respective instrument.
This can now be done during the seven days in which the license remains valid.
Another possible solution for working during NFP would be to synchronize the sequence and method templates with the help of the scheduler in local data vaults provided for this purpose and then use these in such a situation. Unfortunately, the XVault cannot be officially used for this purpose, as the scheduler does not allow synchronization to it. (* see in this chapter)
If the network connection is now re-established, the sequences created during the outage can be removed from the queue. This must be done manually. When removing, the user is asked where on the server these sequences should be saved.
A sequence measured during the NFP cannot be removed from the queue without synchronizing it with the server. All operations can be fully traced back to the individual user, even during a network failure (full traceability). In addition, there is a corresponding note in the preconditions of the instrument audit trail for each injection that this data was recorded during an NFP.
Waters Empower 3.9.0
To bridge shorter network interruptions, a Waters Empower LAC/E (see Glossary) can buffer the measurement series currently being carried out, called Empower Sample Sets (see Glossary) (Buffering Mode).
Waters offers the so-called BC LAC/E (see Glossary) for a surcharge of approx. 700-900 € on the sales price of a normal LAC/E. This offers the option of switching to a so-called BC LAC/E mode, in which the LAC/E becomes a local workstation (separated environment) that can be used without a server infrastructure. However, certain elements such as user information and templates can be synchronized to the local database using the additional tool called Waters SecureSync.
According to the configuration of Waters SecureSync, defined data such as users, user types, user groups, projects and systems are synchronized from the central server infrastructure to the local database of the BC LAC/E (no RAW data). The synchronization can be traced in the audit trail and log files and a corresponding entry is also documented in the Message Center.
Which projects are synchronized to which BC LAC/E can be managed by assigning access groups.
The same applies to users. Only those who belong to a defined user group are also synchronized in the BC LAC/E database.
The information is synchronized separately once a week on each BC LAC/E and must also be configured accordingly on each BC LAC/E via a dedicated XML configuration file (X:Program Files (x86)WatersSecureSyncSecureSyncService.exe.config).
The SecureSync service must then be started and a corresponding synchronization takes place at weekly intervals.
If a network failure occurs, the first thing to do is to wait until all ongoing measurements have been completed. These can only be restored and evaluated once the network is operational again.
To use the BC LAC/E in business continuity mode, all devices and the BC LAC/E itself must be restarted. The WatersService and SecureSync Service must be stopped and disabled.
You can now log on to Waters Empower using the “Local” database.
Waters Empower can then be used as usual, using the local database, although there is a limit to the devices connected to the BC LAC/E and the synchronized users and projects.
Once the network connection has been re-established, it must be ensured that all measurements are completed in business continuity mode. A manual backup of all projects must then be created on an external source, from which they can be manually imported into the Enterprise system. The local projects must be deleted manually from the BC LAC/E. All systems and the BC LAC/E must be restarted. The stopped services WatersService and SecureSync must be enabled and restarted. It must be checked whether the data that was still being recorded during the network failure has been synchronized correctly (recovery log).
Normal operation can then be resumed.
Conclusion
This results in the following evaluation of the responses from Chapter 6:
| Are software licenses and user administration still available if the network fails? | |||
| Will ongoing measurement series continue to be measured without interruption? | |||
| Can measurements be appended to an ongoing measurement series that was started before the interruption? | |||
| Can the data of an ongoing measurement series that was started before the interruption also be evaluated during the interruption? | |||
| Can a new measurement series be started? | |||
| Can existing (validated) methods and report templates still be used? | |||
| Can new measurement series be evaluated and released during an interruption? | |||
| Is data measured during the interruption fully synchronized back to the server | |||
| Are all actions performed during the network outage fully traceable (without manual documentation)? |
Summary
All three manufacturers have a solution to continue operation during a network failure.
This is built into Agilent OpenLab 2.8 by default. However, manual procedures must be added in many areas to ensure complete traceability, as all actions during the outage take place without user management (system user with full privileges). In addition, measurements started before the outage cannot be processed during the outage. After talking to Agilent, it was confirmed that corresponding improvements are already on the roadmap.
Thermo Chromeleon 7.3.2 MUd actually contains everything you need. However, the functionalities are not obvious to the end user and are not documented. Without in-depth knowledge of Chromeleon or appropriate training, it is almost impossible to continue operation. It would also be a great advantage if the scheduler could also be used to synchronize templates in XVault so that they could be used in the NRP. (* see also this chapter)
At Waters, a surcharge must be paid for the BC LAC/E solution so that the users and templates can be synchronized with SecureSync and receive a local license to be able to measure in exceptional cases. Nevertheless, it is not possible to process measurements that were started before the outage during the outage. Restoring the data to the server and removing data from the LAC/E that was generated during the outage is a completely manual process that involves exporting and importing the data. In addition, services must be stopped and started on the BC LAC/E, which requires increased privileges at the operating system level, which in turn represents a risk during normal operation.
Authors
Robin Frey
Robin is a consultant, chromatography data system (CDS) specialist and laboratory information management system (LIMS) expert with extensive experience in the analysis, service and maintenance of laboratory instruments such as GC, MS and HPLC. He brings years of expertise in analytical laboratory and method development as well as the digitization of laboratory processes and specializes in the automation of reporting and processes in chromatography environments. Robin has also trained countless users in chromatography data systems and mass spectrometry as a certified trainer and provides first class support to customers. He is particularly proficient in the use of Thermo Fisher Scientific products such as Chromeleon 6, Chromeleon 7, Xcalibur and Tracefinder as well as the use of HPLC, GC, MS, MSMS and high resolution mass spectrometry.
Piotr Kuchta
Piotr is an experienced Laboratory Systems Administrator and CSV Consultant with a Master's degree in Chemical Technology and a postgraduate degree in Computer Network Management and Design. He has been working in the regulated pharmaceutical industry for over ten years, specializing in the administration and support of laboratory information systems such as Empower, LabX and other systems. He also has experience in analytical instrument qualification and hands-on experience of small molecule analysis in analytical development. Thanks to his combination of technical know-how, analytical thinking and deep understanding of regulatory requirements, Piotr supports laboratory environments in many different aspects.
Lukas Graubner
Lukas is a Senior Consultant and trained chemical laboratory technician with extensive expertise as an Enterprise Chromatography Data System (CDS) specialist. For many years he has been involved in the installation, design, qualification and support of Enterprise Chromatography Data System solutions - from system architecture to the training of users and administrators. He also has extensive practical experience in the support and qualification of HPLC, IC and MS systems. With his deep understanding of GxP, data integrity and 21 CFR Part 11, Lukas ensures compliant, stable and future-proof laboratory IT solutions - especially in the environment of Thermo Fisher Scientific Chromeleon 6 and 7.
wega is a vendor-neutral service provider that supports laboratories in the selection and implementation of suitable software solutions. With experienced, trained employees, wega offers a comprehensive service package for the systems compared in this article - Agilent OpenLab CDS, Waters Empower and Thermo Scientific Chromeleon. The services range from strategic IT consulting and process analysis to the conduct of tenders through to system provision and configuration. In addition, wega provides support with computer system validation, project management and organizational change in the course of digitalization. The range of services is supplemented by method and report development, support contracts (SLA), interface integration and customized training for users - always with the aim of ensuring stable, efficient and compliant laboratory processes.
As part of such a service, *wega can also configure the synchronization of methods and sequence templates in XVault (see this chapter)
Glossary
| Activity Log | Activity Log | Data and system audit trail from Agilent OpenLab |
| AIC | Agilent Instrument Controller | Client computers to which the chromatography instruments are directly connected within Agilent OpenLab CDS 2 |
| BC LAC/E | Business Continuity Laboratory Acquisition Control Environment | Is a product with a surcharge to the normal LAC/E to use it as a standalone client in exceptional cases |
| Data Vault | Daten Container (Tresor) | Thermo Fisher's brand name for the top level of the data structure which consists of a database and associated RAW files |
| IPC or IC | Instrument Controller PC | Thermo Fisher's name for the PC which controls the instrument |
| LAC/E | Laboratory Acquisition Control Environment | Is a brand name of Waters for the standardized instrument controller PC distributed by Waters |
| NFP | Network Failure Protection | Thermo Fisher's solution for business continuity in the event of network or server failures in Chromeleon |
| Sample Set | Measurement series or sequence | Waters naming for the composition of a measurement series |
| Scheduler | Thermo Chromeleon Scheduler | A scheduler service in Chromeleon which enables the execution of time-controlled actions, such as replicating templates |
| Sequence | Measurement series or Sample Set | Agilent and Thermo naming for the compilation of a measurement series |
| XVault | Hidden Data Container (Vault) | Thermo brand name for the hidden Data Vault which caches the data during recording |
Source References
Agilent:
- Agilent OpenLab CDS - Acquisition Failover Users Guide (Edition: 02/2025)
Thermo:
- Thermo Scientific Product Spotlight - Ensure Business Continuity No 73781 (2020)
Waters:
- Empower Business Continuity LAC/E SecureSync - Installation and Configuration Guide (Doc No. 715008070 Version 01, Feb 2023)
- Empower Business Continuity LAC/E SecureSync 2.1.0 - Release Notes (Doc No. 715008381 Version 00, Dec 2022)
- Webinar - Informatics series 04 - Business Continuity & the BC LACE of 24. May 2023 held by Carsten Fischer
Trademarks
Empower™, LAC/E™, SecureSync™, Waters™ are trademarks of Waters Corporation.
OpenLab CDS™ is a trademark of Agilent Technologies
Chromeleon™ is a trademark of Thermo Fisher Scientific Inc