NVIDIA FLARE Details New Workflows for Federated Multimodal AI and VLMs

Vision-language models can already answer questions about an image, generate captions, and reason across pictures and text. But the data needed to make these models useful in specialized fields — medical scans and radiology reports, financial documents and transaction records — often can’t leave the walls of the institution that holds it. Privacy rules, competitive concerns, and data-sovereignty requirements keep it locked in place.

NVIDIA’s federated learning team has just laid out, in detail, how its open-source FLARE framework is meant to handle that problem for multimodal AI specifically. In a technical blog post published August 19, 2026, NVIDIA engineers Ziyue Xu, Holger Roth, Zhihong Zhang, and Peter Cnudde describe the design choices involved in training vision-language models across institutions that can’t pool their raw data — and point to a concrete research result, called FedUMM, that shows just how much more efficient this can be than previously assumed.

What NVIDIA Announced

The post isn’t a product launch. It’s an engineering explainer built around two questions: what model information should actually travel across the network during federated training, and how should that information move and be combined once it does.

NVIDIA FLARE — short for Federated Learning Application Runtime Environment — is the open-source Python framework the company uses to answer both questions. It lets multiple sites train a shared model without sending their underlying data to a central server; instead, each site trains locally and only model updates travel outward. The new post applies that idea specifically to vision-language and unified multimodal models, and showcases FedUMM, a research collaboration between William & Mary and NVIDIA, as its central example.

Why Federating Vision-Language Models Is Hard

Centralized training is comparatively simple: gather images, captions, and question-answer pairs into one pipeline and train against all of it. Federated training breaks that assumption. Different sites may hold different mixes of images, text, and other data, and the model updates they generate can be large — sometimes large enough to overwhelm both network bandwidth and server memory.

That creates two separate engineering problems. First, a federated system has to define what each site is allowed to train and share — some sites might work with different modalities or tasks, so the rules for combining their updates need to be explicit. Second, if a workflow sends full model weights back and forth, those updates can be enormous, and a server aggregating updates from many clients at once has to hold all of them in memory simultaneously.

Researchers have explored different ways around this. Some approaches, such as one called CreamFL, exchange distilled knowledge instead of raw model weights. Others, including FedCLIP, FedPIA, and FedUMM, freeze most of a pretrained model and only train and exchange a small set of add-on parameters. NVIDIA FLARE is designed to support either strategy — full-model updates or lightweight ones — depending on what a given workflow needs.

The Engineering Bottleneck: Bandwidth and Memory

For workflows that do require larger updates, FLARE includes three specific mechanisms aimed at the bandwidth-and-memory problem:

• Large-object externalization replaces bulky data inside a network message with a lightweight reference, transferring the actual payload separately so the control message itself stays small.
• The FLARE Tensor Downloader streams PyTorch model tensors incrementally, using a pull-based approach so only the requested piece of data is serialized and moved at any given moment — reducing peak memory during distribution. TensorFlow workflows currently rely on a more traditional serialization path instead.
• Disk-backed aggregation, introduced in FLARE 2.8.0, writes incoming updates to temporary files on disk rather than holding every client’s update in memory at once, preventing memory use from growing in lockstep with the number of participating sites.

These are platform-level features, built into FLARE itself, and they matter most for workflows that can’t rely on lightweight adapters alone.

FedUMM: Federating Adapters Over a Frozen Backbone

The article’s central example takes the opposite approach: minimize what crosses the network in the first place. FedUMM, developed by researchers at William & Mary in collaboration with NVIDIA and supported through NVIDIA’s Academic Grant Program, keeps a frozen BLIP3o multimodal backbone at each participating site. Rather than retraining that entire backbone, each client trains only small LoRA (low-rank adaptation) adapters locally. FLARE then coordinates the training rounds and aggregates just those adapter updates — not the full model.

The project is designed with generality in mind, with separate encoders envisioned for vision, audio, and text, though the experiments described so far focus specifically on vision-language tasks. The work was recognized with an Outstanding Student Paper Award at the FL@FM workshop at TheWebConf 2026.

It’s worth being precise about what FedUMM is and isn’t. It’s an academic research project supported by NVIDIA, evaluated so far using simulated clients and synthetic data splits — not a commercial NVIDIA product, and not yet tested against real multi-institutional data.

The Numbers: Communication Savings and Accuracy

The efficiency gains reported for FedUMM are substantial. In an eight-client comparison, the researchers found that exchanging only adapters — rather than full model weights — cut per-client communication from 28.6 GB to 0.094 GB per training round, roughly a 300-fold reduction. On the VQA v2 visual-question-answering benchmark, this adapter-only approach also scored 0.7 points higher than a full-model federated averaging baseline. And at eight clients, FedUMM’s performance stayed at roughly 97 percent of what a centralized (non-federated) training run achieved on both VQA v2 and the GenEval compositional-generation benchmark.

The researchers tested up to 16 clients under varying degrees of data heterogeneity (using a statistical method called Dirichlet partitioning to simulate realistically uneven data distributions across sites), and found performance degraded slightly as the number of clients and the degree of heterogeneity increased — though it remained broadly competitive with centralized training throughout.

These figures come from the research team itself — NVIDIA and William & Mary — rather than from an independent third-party benchmark, and NVIDIA is explicit that the evaluation doesn’t establish clinical-grade performance or formal mathematical privacy guarantees. What it does show, in the authors’ own framing, is that raw training data can stay local throughout the simulated workflow while the model still learns effectively.

Figures reported by NVIDIA and William & Mary researchers based on simulated experiments; not independently replicated.

Not a Pre-Built CLIP or LLaVA Toolkit — What’s Actually in the FLARE Repository

Given how often CLIP and LLaVA come up in discussions of vision-language models, it would be reasonable to expect NVIDIA FLARE’s public code repository to include ready-made examples built around them. Based on available documentation and the repository’s own research index, it doesn’t appear to.

What is actually present are two specific multimodal examples. The first is FedUMM itself, built around the BLIP3o backbone described above. The second, documented separately in an NVIDIA blog post about an “Auto-FL” research-automation tool, is a federated Qwen3-VL LoRA training workflow that simulates three medical imaging sites — using datasets known as VQA-RAD, SLAKE, and PathVQA — and evaluates results with a token-level F1 score. That example, too, is a simulation rather than a real multi-institutional deployment.

This isn’t a claim that CLIP or LLaVA support definitely doesn’t exist anywhere in the codebase — a full repository is large, and this review wasn’t exhaustive of every branch or pull request. But no such example turned up in the documentation reviewed for this article.

Where This Fits in NVIDIA FLARE’s Evolution

NVIDIA FLARE didn’t start as a standalone product. Its federated learning capability originated inside NVIDIA’s Clara Train software for medical imaging, before being spun out in 2022 as an open-source, domain-agnostic framework that could be used well beyond healthcare.

Since then, the framework has picked up broader capabilities. In 2024, FLARE 2.4.0 added support for training large language models, including techniques like prompt-tuning and parameter-efficient fine-tuning, working with NVIDIA’s NeMo framework. More recently, FLARE 2.7.x introduced a substantial memory-management overhaul across the server and client stack. FLARE 2.8.0 — the version referenced throughout the new multimodal post — brought the Job Recipe API (a simplified way of defining federated learning jobs) to general availability, alongside the FedUMM and Qwen3-VL examples, Docker and Kubernetes deployment options, and a new differential-privacy example tied to a financial fraud-detection use case. A newer patch release, 2.8.1, is available through the Python Package Index as of this writing.

It’s worth noting that differential privacy itself — a mathematical technique for limiting how much information a model update can reveal about the underlying data — isn’t new to this release. FLARE has supported differential-privacy filters since at least version 2.2. What’s new in 2.8.0 is additional example code demonstrating it, not the underlying capability. Separately, while FLARE has documented ties to NeMo for language-model work, the new multimodal/VLM post itself doesn’t describe this specific workflow as running through NeMo or Clara — its models (BLIP3o, Qwen3-VL) come from the broader open-source ecosystem instead.

Real-World Use: How Far Along Is This?

It’s important to be clear-eyed about where this work currently stands. Both FedUMM and the Qwen3-VL medical example are demonstrated using simulated clients and synthetic or benchmark datasets — not live data shared across real hospitals, banks, or research institutions.

NVIDIA does state, more broadly, that organizations including hospitals, national laboratories, and financial institutions use FLARE in production today. That’s a general company claim about the framework as a whole, not a confirmation that this specific multimodal workflow is running anywhere outside a research setting. Separately, a healthcare-focused federated learning company, Rhino Health, integrated FLARE into its own platform several years ago for research into diagnosing brain aneurysms — a real, if unrelated and unimodal, example of FLARE being used with actual institutional partners.

How NVIDIA FLARE Compares to Other Federated Learning Frameworks

FLARE isn’t the only federated learning framework available, and NVIDIA’s post positions it within a broader field that includes Flower (from the company Adap), Google’s TensorFlow Federated, OpenMined’s PySyft, and IBM’s federated learning tools.

The relationship with Flower is more layered than straightforward rivalry: NVIDIA FLARE actually includes a ready-to-use “Flower” job recipe, allowing the two frameworks to work together in some configurations rather than purely competing. NVIDIA describes FLARE’s broader differentiation as its production tooling — secure certificate-based provisioning, per-site authorization policies, audit logging, and support for hardware-based confidential computing. That’s a claim NVIDIA makes about its own platform; no independent, side-by-side benchmarking of FLARE against Flower, TensorFlow Federated, PySyft, or IBM’s offering — specifically for multimodal or vision-language workloads — appears to be publicly available.

Limitations and Open Questions

Several things remain genuinely unresolved. The efficiency and accuracy figures for FedUMM haven’t been independently reproduced outside the NVIDIA/William & Mary team. No confirmed production or clinical deployment of this specific federated VLM workflow has been identified — everything described is simulation-based so far. NVIDIA’s own researchers note that the evaluation doesn’t establish formal privacy guarantees, meaning “federated” here should not be read as a synonym for “provably private.” And there’s no publicly available data comparing FLARE’s approach to competing frameworks on comparable multimodal tasks.

Latest Developments

As of this writing, FLARE 2.8.1 is available as a patch release through the Python Package Index, following the 2.8.0 release that introduced the FedUMM and Qwen3-VL examples described above. No further NVIDIA announcements specific to federated multimodal or VLM training have surfaced since the August 19 post.

Conclusion

The practical takeaway from NVIDIA’s latest FLARE research is fairly specific: when a federated learning workflow can get away with training only lightweight adapters on top of a frozen multimodal model, it can cut communication costs dramatically — by roughly 300-fold in FedUMM’s reported experiments — while sacrificing very little in the way of model quality. That’s a meaningful engineering result for anyone designing systems meant to train on data that legally or practically can’t be centralized.

What it isn’t, yet, is proof that this specific approach works at clinical or institutional scale, or that it satisfies formal privacy requirements on its own. The examples described so far run in simulation, using synthetic data splits and public benchmarks. Whether FedUMM or similar adapter-based federation moves from research demonstration into real multi-institutional pilots — in healthcare, finance, or elsewhere — is the detail worth watching for next.

FAQ

What is NVIDIA FLARE? NVIDIA FLARE (Federated Learning Application Runtime Environment) is an open-source Python framework that lets multiple organizations train a shared machine learning model without centralizing their raw data. Each site trains locally, and only model updates are exchanged with a coordinating server.

What is federated learning and why does it matter for vision-language models? Federated learning coordinates model training across separate data-holding sites instead of pooling all data in one place. It matters for vision-language models because the image, text, and document data needed to train them for specialized domains — like healthcare or finance — is often legally or practically restricted from leaving the institution that holds it.

What is FedUMM and who developed it? FedUMM is a federated learning framework for unified multimodal models, developed through a collaboration between William & Mary and NVIDIA and supported by NVIDIA’s Academic Grant Program. It trains lightweight LoRA adapters over a frozen BLIP3o multimodal backbone, with NVIDIA FLARE coordinating the federated training process.

How much does adapter-based federation reduce communication costs compared to full-model training? In an eight-client experiment, FedUMM’s researchers reported cutting per-client communication from 28.6 GB to 0.094 GB per training round — roughly a 300-fold reduction — while modestly improving accuracy on the VQA v2 benchmark compared to a full-model federated averaging baseline.

Is NVIDIA FLARE’s federated VLM training being used in production today? Not based on available evidence for this specific workflow. FedUMM and a related Qwen3-VL medical imaging example are both demonstrated using simulated clients and synthetic or benchmark data. NVIDIA states more broadly that hospitals, national labs, and financial institutions use FLARE in production, but that is a general claim about the framework, not confirmation that this particular multimodal workflow is deployed outside research settings.

Does NVIDIA FLARE include pre-built examples for CLIP or LLaVA? No such example was found in the FLARE repository’s documented research examples. The multimodal examples that do exist are FedUMM (built on a BLIP3o backbone) and a Qwen3-VL-based medical imaging workflow.

How does NVIDIA FLARE compare to other federated learning frameworks like Flower or TensorFlow Federated? FLARE competes with frameworks including Flower, TensorFlow Federated, PySyft, and IBM’s federated learning tools, though its relationship with Flower also includes some interoperability through a built-in Flower job recipe. NVIDIA describes FLARE’s production-oriented security and deployment tooling as a differentiator, but no independent benchmarking against these alternatives for multimodal workloads was publicly available at the time of writing.