FHIR data sync patterns every developer needs in their toolkit

A number of the FHIR implementations I see run into the same problem: people load everything into the server with no checking on the way in, and end up with millions of resources where there are only about thousands of different resources actually in it.

That mistake is what I see driving cost, performance problems, and duplicate-riddled data. The reason is simple: your FHIR server is a data access layer, not an ETL (extract, transform, load) engine. Running ETL work through it adds overhead to every request that goes through afterward.

There is no single best way to load data into a FHIR server. It depends on your use case. But there is a best way to think about it, and it starts with keeping the transformation work out of the server.

Stage first: the case for a database in front

I was an SQL engineer doing a lot of ETL data work and quality reporting, so this is an old instinct, but it holds even more for FHIR: put a staging database between your source systems and your FHIR server to track changes and sync differences only.

Stage it in a database and you can deduplicate, see what changes have been made, compare things, and hash it out, then load only the records that changed.

Your FHIR server is a data access layer. Every write through its REST pipeline pays for validation, indexing, and security. That is what you want when the data comes from outside and you need to confirm the client can write at all. That is less necessary when the data is coming from a source of truth you control, with security based on being inside your own network.

With staging in place, four variables decide how you move the data in: volume, acceptable latency, how much security you need on top of the data, and the source system. Can it be thirty seconds late, or does it have to be instantaneous? Four patterns cover most requirements.

Pattern 1: the REST API

The straightforward option: standard FHIR creates, updates, conditional updates, and transaction bundles, through the client SDKs available for .NET, Java, Python, and most popular languages.

For a few hundred or a few thousand resources at a time, the REST API is perfectly capable, and it is the right choice when your use case needs the full pipeline: security enforcement, validation, server-side logic. Application-driven updates belong here.

The trade-off is that the API shares server capacity with the clients trying to read and search, so push a big load through it and those clients wait longer than they expect. Compute is the other cost, because the API does a lot of work in the back end and CPU and memory ramp up.

It is also where the duplicate problem bites. As an example, I have seen it a number of times where an organization has a provider directory, and the same providers get loaded in with no changes, over and over, sometimes multiple times a day. Conditional updates help, and some servers skip the write when the resource has not changed, so you do not get another history entry. But a staging layer that never sends the duplicate in the first place helps more.

Pattern 2: bulk import

For initial loads of millions or billions of records, bulk import is the serious option, and most commercial FHIR servers support one. These tools go direct to the database or data store. Ours, Firely Server Ingest, reads a JSON or NDJSON (newline-delimited JSON) file and batches a thousand entries at a time per second or more, depending on configuration.

Because it bypasses the pipeline, you need something upstream that validates everything first. Most FHIR servers have an indexing strategy, so writing direct to the database means matching that strategy if the load is going to stay fast.

The other risk is contention, since at these volumes the load is heavy on the database and competing with live traffic. The common pattern is to bulk load at off hours like early in the morning at 2AM, and either take the API down temporarily or at least run it when it affects fewer users and clients.

Pattern 3: PubSub and event-driven sync

Once the initial load is done, the question is keeping the server synced with the source. Enable change data capture on the database, poll every thirty seconds or the acceptable latency time frame , and generate a message onto a queue. RabbitMQ, Azure Service Bus, and Kafka are the brokers we work with most. Your FHIR server subscribes and processes the messages into creates and updates.

This suits a few hundred or a few thousand changes an hour, where a few seconds of latency is acceptable. It will usually bypass authorization and validation, which is what makes it much faster than the API, so it fits when the source data is already trusted and clean.

The control knob is the consumer count. Maybe you want ten messages processed at the same time. Much more than that can impact server performance and slow down other clients, so it depends on where your server is deployed.

One tooling note, and it is not what I am trying to sell: MassTransit is what we use to abstract the message bus, so you are not coding the message handling yourself. It stopped being free at v9, so factor that in.

Pattern 4: the facade

The facade always seems like a great idea when you are implementing it. Serve FHIR straight from your existing database, no migration. Everybody’s favorite thing to ask for.

The problem is that you have to write everything yourself. Your FHIR server does not know how to search a proprietary schema, so you have to build a translation layer underneath that goes right to the database and applies all the appropriate search features of FHIR, and those layers tend to be very chatty.

We work in .NET, so people reach for Entity Framework and see dozens of queries for a single entry. Whatever effort you saved by not engineering ETL, you will probably spend more of making the facade efficient and mapping data to the implementation guides you need.

Keeping it supported is the other cost. A facade usually depends on the same libraries as the FHIR server, so every upgrade and every search parameter change becomes yours to keep in sync. And facades generally do not keep history: if the data changes in the source system, the previous state is gone unless the source was designed to retain it.

Sometimes you do not have a choice and the facade is the best option. EHRs are the classic case, which is also why you sometimes see performance problems on EHR bulk exports: the request goes into a back end that was never optimized for FHIR delivery, so everything is often mapped on the fly.

For simple gets and simple searches, a facade is not a bad idea at all. It is what you do when you cannot sync. Just make the choice early, before you have invested a lot of engineering and then find out the performance does not work for you.

Choosing, and combining

Side by side, the patterns trade off across the same handful of things: complexity, performance, cost, maintenance, data freshness, and pipeline enforcement.

• REST API: lowest complexity, since it already exists and all you need is a client. Full pipeline enforcement, real-time freshness, but the most expensive per record.
• Bulk import: the fastest by far, lowest infrastructure cost, but no enforcement, which is why everything has to be pre-validated first.
• PubSub: medium complexity, near real-time freshness, and configurable enforcement, so you decide what runs on each message.
• Facade: real-time freshness, but the highest development and maintenance cost of the four, since every search and every server upgrade is your problem.

If you need security, validation, and indexing out of the box, that is the REST pipeline’s job. Anywhere else, you are building those steps yourself.

The important thing is that you can use a hybrid approach, and most mature architectures do: a bulk import once a night for everything that changed during the day, with REST or PubSub keeping things current in between. It depends on how fresh your data needs to be.

If you are weighing these options, this is the conversation we have with teams adopting Firely Server, and staging is always where we start. If you want to talk through the right pattern for your data, get in touch.