Who The Heck Designed This Biopharm Plant?
In the first two parts of this series, we covered production operations and maintenance of biopharma facilities, and we exposed the most common flaws in a plant.
Now, what if a plant was saddled with a bad design driven out of inexperience or cost control? In this situation, the plant must adapt as best as it can, given the process or facility it was handed. We will explore several common design mistakes we have seen and hope you recognize them and prevent their recurrence.
Over the years we have had the privilege of working in a wide spectrum of biopharma plants, from mAbs to vaccines and from aseptic filling to cell therapies. We are always bringing the processes online, solving process problems, and optimizing the production. In these facilities, we often bang our heads over built-in “features” and see a lot of questionable design decisions that complicate operations and ultimately cause disruptions to production. In some cases, the design decisions were driven by either cost cutting or a lack of understanding of operations.
Often, as we are working on problems in the plant, we ask the operators if they were consulted during the design or if they were even trained on the equipment. In almost all occasions, we find the answer to these questions is “no,” and the operators are just as perplexed as we are. In the end, we all must adapt and work around the design flaws to make the operation work and get production running.
The following is a discussion of these critical flaws in design or, perhaps, flaws in design decision-making. These are broken down into:
- general or large-scale decisions that impact the entire process or facility and
- specific, bone-headed design and construction flaws that impact part of a process or introduce a weakness in a processing suite.
General Biopharma Design Mistakes
The first major area of design concern is process scale as it relates to vessel sizing and risk. We have seen a wide spectrum of designs ranging from trains of 10 x 500-liter stainless-steel bioreactors to an entire building and entire product line centered around one large-scale 10,000-liter stainless-steel bioreactor. When working on designs we need to consider a few aspects:
Risk of process failure
This is especially serious in the early stages of startup, where process failures are common and having one train or vessel will delay the product launch. Multiple trains or multiple vessel configurations provide for more data on startup conditions and reduce the catastrophic nature of a single failure. In addition, with multiple smaller trains we will have a greater opportunity to use affordable single-use systems (SUS). Using a basic production model and risk assessment, two to four parallel processes make for a prudent design in general, rather than using one vessel.
Overestimating the capacity of single-use
SUS process vessels do have a practical limit today, with the largest commercially available unit being 5,000 liters. These, like many other SUS, have had supply chain shortages, thus the risk related to acquiring this size of vessel is real and can be costly. In this case, multiple smaller SUS reactors can reduce several aspects of risk.
Underestimating the validation effort
On the opposite end of the spectrum is startup time for production and the need to validate multiple (even identical) trains. As we reduce our risk and increase the number of process vessels and trains, we require more staffing and time to commission, qualify, and validate the processes, thus delaying product introduction into the market but assuring some level of production by reducing the risk profile.
Single-Use Or Stick With Stainless?
The complementary area to the discussion of scale is the use of SUS process vessels vs. stainless-steel vessels. We believe the use of SUS is a major advantage due to the elimination of cleaning and cleaning validation, ease of use by technicians and operators, dramatically lower capital costs, lower utility costs (e.g., CIP/SIP), and a smaller physical footprint that allows the facility to be smaller and HVAC costs to be lower. This becomes a major advantage for multi-product facilities and contract manufacturers (CDMO/CMO).
Given that we are advocates of SUS, based on what we have seen in the last five years, we have shifted our thinking to supporting the use of stainless-steel vessels in the design, given a few caveats. These caveats have evolved recently due to the practicality of supply chain disruption/delivery, increased use of ethanol for liposomes, and the cost of the larger SUS. The following are some factors that support the use of stainless steel:
- The target process vessel is greater than 1,000 liters.
- The process uses any solvents, surfactants, or aggressive materials that would cause greater extractables or leachables in the SUS.
- The manufacturing process train is dedicated to one high-volume product only and cleaning has been automated and optimized. In this case, the automated CIP may be able to turn the vessel over faster than a SUS changeover for higher capacity, combined with a long-term cost benefit. As an example, a seasonal vaccine production could be a great candidate for this.
- The process is simple, and a long-term capacity commitment was made to such production as IV bags, blood processing, and certain mAbs. In this case, with cleaning/cleaning validation solidified, the cost reduction favors a higher volume.
Respect The Space
Gray space vs. white space is the design concept that puts all the stationary process equipment (stainless steel vessels, piping, pumps, CIP skids, utility skids, etc.) in a non-classified mechanical space or gray space.
The only equipment and piping exposed in a classified space (white space) are what the operator must manually interface with to change out the SUS, access sample ports, connect additives/inoculation, and swap out the frequently changed filters. This minimizes the classified space, reduces the HVAC installation and cost, and presents a very consolidated process workspace for the operator.
Building classified facilities today around large static equipment makes absolutely no sense and drives up the capital and operating costs. In addition, the vessels themselves, the piping, valves, instruments, conduit, and support gear are generally uncleanable on the exterior and provide a surface for the collection of dirt and a variety of microorganisms. To compound the problem, these vessels and processes need high bay classified rooms to house them, and they in themselves cannot be cleaned properly (ceilings and walls).
In facilities that heavily depend on SUS, this concept is more challenging to implement since the expectation is that single-use connections will be made within a classified background. Thus, we need to limit the approach to hybrid systems with the SS portion in gray space or media/buffer totes in gray space that feed the process in the white space via tubing pass-throughs.
A Case Study On The Woes Of Process Isolation
Process isolation for media preparation is another area where we see many transgressions. Several years ago, we were diagnosing a problem in the purification suite of a large-scale biopharma facility on the East Coast. We were baffled by the high environmental monitoring (EM) “hits” in a generally uncomplicated suite.
The root cause of the problem was that one of the columns was loaded in the upstream media prep suite and then pushed through the corridors and into the purification suite (this also violated previral vs. post viral separation). This also explained the contaminations in the corridor and adjacent spaces.
The media prep was done with drum dumpers, open vessels, media dust, and powder everywhere, including on the decking, equipment surfaces, operators, clipboards, and everything in the suite. Media prep suites and media handling must be thoroughly designed.
All media transfers need to utilize SUS (e.g., DoverPacs) with a tri-clamp collar secured and a butterfly valve to open/close the media flow. When undocking the tri-clamp collar, the vessel collar, the SUS interface, and the area around the transfer must be vacuumed with a HEPA-filtered vacuum. The discarded SUS plus the operator’s apron must all be placed in a plastic container and sealed before they are removed from the suite. All these aspects of process, operator, and logistics need to be designed in, especially regarding moving media in and out.
Ditch Transfer Panels And Do This Instead
Transfer panels seem to have been the trend starting in the mid-1990s and, somehow, we have never shaken that flawed design concept from our business. In the olden days to correctly route our liquid product material, WFI, or CIP fluid, we built a phalanx of open pipes with tri-clamp fittings, a fluid catch basin, gaskets, clamp-on safety valves, and a book of operator instructions for each route. Transfer panels are a mistake waiting to happen, whether it is a missed connection, dumped product, hot CIP fluid in the basin, EM hits, or cross-contamination from gasket reuse.
It is 2024, and the preferred alternative to transfer panels are the various automated transfer systems (PLC driven) with aseptic and no-mix valve systems (e.g., GEA Varivent, Aseptomag, or other mix-proof systems). These provide rapid, automated switching across product, WFI, and CIP piping, which prevents cross-contamination. Within a month of use, these systems will pay out more than any savings the primitive manual system could provide. The automated valving systems provide a capacity increase via faster batch turnover with repeated CIP applications, in addition to lower EM hits from the removal of the splash basins and manual handling.
Individual Or Specific Considerations
Avoid manual cleaning whenever possible
Why we still have manual cleaning of smaller vessels amazes us. There are automated parts washers of all sizes and services (Belimed, Girton, etc.) and CIP skids that hook up directly to vessels for clean out of place (COP). Ultimately, all stainless-steel vessels will have to pass-through cleaning validation, which requires time and consistency that is not possible with manual operations. Yet another good application is to eliminate the small stainless-steel vessel and replace it with SUS.
Avoid silicone hoses, too
Another scourge of cleaning, and sometimes product transfer, is the liberal use of flex hoses or bare silicon hoses. Rather than the transfer panels mentioned above, we have seen designs that use flexible hoses to connect various process elements as a matter of procedure. These hoses must be cleaned, dried, and stored after each use and consequently become a major area for wear, tear, and misuse, as well as a microbial breeding ground, not to mention cross contamination.
A major pharma lost $100 million in biological production due to the use of a contaminated $500 hose. In each misaligned design, we put the burden and blame on the operators. We recommend hard pipe for all transfers. Use automation and employ simple automated diaphragm valves.
Outlaw dirty transfer practices
We see plants today that have open vessel transfer with shovel-based plastic Home Depot buckets and other miscellaneous methods of open conveyance to drag ingredients around. Handling of solid material additives for salts, buffer powders, NaOH, lipid powders, etc., should be via sealed SUS (like the media mentioned above) and attached to the vessel for easy and low bioburden transfer. This is a perfect example of designers leaving the burden (maybe bioburden) to the operating staff.
Liquid additives should be treated in a similar way to the solids if they cannot be pumped in from a central reservoir. Liquids like HCL and NaOH should never be hand pumped into open containers and dispensed manually. This is a fundamental error in the thought process as it violates safety practices. All these pH correction liquids can be purchased directly from the vendors in SUS.
Don’t skimp on the floor drain covers
Never design floor drains into an EU grade C room, and avoid it in EU grade D. If drains are needed for process changeout, all drains should be fitted with a gasketed, stainless-steel cover plate.
Avoid infrastructure overkill
One facility that we had the pleasure of doing a design review designed in 21 WFI drops for janitorial closets! When reviewed, the piping and valving had cost several hundred thousand per drop, all for about 10 liters of water, each ranging from one to three times per day. This expenditure of capital is wasteful, and the periodic sampling/sterility testing of all these taps is an unacceptable recurring cost.
We recommend buying compact totes of pre-made sanitation liquid from the manufacturer and buying WFI in totes/bags/bottles for the rinse solution. These all can be stored in the janitorial closet and replenished as needed.
Check your clearance
One of the more comical design flaws recently (2021) was a 1,000-liter stainless-steel vessel erected in a mechanical space. The top swing hatch on the vessel was 18 inches in diameter but the ceiling over the tank was 12 inches from the hatch. Thus, the hatch could never be opened for inspection. Somehow, the architect and the process designer never met one another to verify clearances.
In the same mechanical space as the ill-fated swing hatch above, the valving, piping, and instruments for a process were preassembled off-site and “layered in” the process space. The problem is the valving and piping network looks like the engine compartment of your car and requires substantial process disassembly to replace an instrument. In this case, a piping designer needs a lesson in plant maintenance.
Likewise, don’t skimp on the steel
Another specification mistake that saves some money on paper calls for all the handrails, staircases, and kick plates to be made from low-grade steel (304 or lower). This seems to have been a theme in one facility that turned a process suite into the “rust suite” after a few months of treatment of Spor-Klenz and other cleaning agents.
Say no to threads
Fittings specifications would seem a relatively simple item for a biopharma facility; however, when not done completely, the construction team will use whatever is in their toolbox. There should be no threaded fittings on equipment, utility lines, conduit, or power connections. We do not need any aspects of the facility that are difficult to clean.
Skip the pits; use flush-mounted scales
As recently as 2022, an engineering firm was recommending the installation of scale pits for tote weighing inside an EU grade C room. Scale pits are a notorious collection point for dirt, debris, and microorganisms and are a challenge to clean. Instead, install low-profile, flush-mounted electronic scales, which cost 20% of scale pits. They have elongated ramps to climb the ¾-inch height and can be flipped up for cleaning the floor.
Since we discussed scales, if we need to use a weigh booth it is critical that we have the airflow patterns correct to keep the dispensing operation a low bioburden event. The only way to ensure that we have the critical air contours correct is to buy a pre-made weigh booth from a vendor. We can never assume we can jam a few HEPAs in a corner and that will be good enough, as it will not work.
Mind the background
If we are building a QC lab with sterility testing isolators, we cannot assume an office space with office type finishes and HVAC will be fine when we put a high-quality isolator in the middle of it.
Experience has shown that to prevent inadvertent contamination from entering the isolator you need to build the background room with the same quality and HVAC as an EU grade C suite. The consequence of not doing that is the occasional false positive on sterility samples, which will create deviations and investigations that will interrupt production.
Rein in the sample taps
Within every biopharma facility, there will be many WFI drops, and they all need to be sampled to ensure our WFI quality is appropriate. When designers make allowances for sample ports, they rarely specify the installation criteria, and this leads to 20-foot lengths of ¼-inch tubing snaking around the walls and ceilings. These sample taps ultimately build up a layer of rust and a variety of microorganisms and typically will provide a positive on sterility when sampled.
As a rule, sample lines must be less than 1 foot, ½-inch stainless steel, and completely vertical.
Keep your connections away from drains or risk catastrophe
The last of these observations comes from when a CIP hose was left hanging from a process piping system into an open drain. Amid the routine heating and cooling, flowing, and stopping, this plant created a backflow from the drain into the process lines that caused a catastrophic bioburden issue. Never provide any connection from a hose or pipe to a drain without it specifically being a drain line.
Conclusion
These observations and recommendations come from a history of watching plants and systems fail because of what they were given by their engineers and designers. These notes should be a reminder that all designs need to be practical and operable by people. Sometimes, we need to step back from the drawings and calculations and think about what we have done and if it makes sense. All the above examples are real, and the engineers thought what they were doing was correct, but common sense needs to be applied. This common sense is where you must employ the wisdom of the operators and maintenance technicians. Do that, and always ask yourself this following:
- Is this safe?
- How do I clean this?
- Does the layout of the process that the operators will use follow a logical unidirectional path?
- How would I get at this to repair it?
- Can I eliminate a tedious or manual task here?
In conclusion, designing and engineering a biopharma plant is a process of conversations with all the stakeholders who must work in what we build. It requires thought and time.
About The Authors:
Herman F. Bozenhardt has 48 years of experience in pharmaceutical, biotechnology manufacturing, engineering, and compliance. He is a recognized expert in aseptic filling facilities and systems and has extensive experience in the manufacture of therapeutic biologicals and vaccines. His current consulting work focuses on aseptic systems, liposomes, biological manufacturing (BL-1, BL-2, BL-3), and automation/computer systems. He has a B.S. in chemical engineering and a M.Sc. in system engineering, both from the Polytechnic Institute of Brooklyn (now NYU). He can be reached via email at hermanbozenhardt@gmail.com and on LinkedIn.
Erich H. Bozenhardt is the associate director of process engineering for Untied Therapeutics in Raleigh, North Carolina. He has 18 years of experience in biotechnology and aseptic processes and has led several biological manufacturing projects, including cell and gene therapies, mammalian cell culture, and novel delivery systems. He has a B.S. in chemical engineering and an MBA, both from the University of Delaware. He can be reached via email at erichbozenhardt@gmail.com and on LinkedIn.