CURDNet: Contrastive Ultrasound Report Generation with Diversity-Aware Learning

Blog · Aug 2025 · Ultrasound report generation

TLDR

In this blog we investigate the difference between ultrasound images and x-ray images, and try to train a model for ultrasound report generation.

Introduction

Ultrasound is used across many organs, and the image appearance depends heavily on the operator, view, and clinical setting. That makes the data much less uniform than many chest X-ray benchmarks. A model that performs well on one organ can therefore struggle when the next case looks completely different. As shown in the figure at right hand side, the X-ray image pairs are relatively similar, difference are mainly focused on local; while the ultrasound image pairs are more diverse. So we come up with the idea, instead of modify the transformer model to focus on local difference between different cases, we want the model to identify differences between cases level -- we bring a CLIP task into training a image caption model.

Comparison of X-ray and ultrasound data diversity — Ultrasound cases vary more across organs and acquisition settings than the standardized views common in chest X-ray data.

Method

CURDNet combines three lightweight ideas:

EchoDice. We come up with the idea to use a very simple but effective strategy. Instead of training three distinct models for respective subsets, we use one model to train all samples, with high within-batch diversity being able to, encourage the model to learn across organs instead of memorizing one visual style.
ReportMatcher. This is the CLIP module adds image-report contrastive learning so matched pairs move closer and mismatched pairs move apart.
ReportJudger uses an LLM-based scorer to check whether retrieved reports are clinically and semantically relevant, in order to tell whether our encoder actually learns the case difference.

The system trains a shared model for mammary, thyroid, and liver ultrasound reports rather than maintaining one model per organ.

CURDNet model architecture — CURDNet combines diversity-aware sampling, report-image matching, report generation, and an LLM-based relevance judge.

Key results

On the reported benchmark, the full model achieved BLEU-1 scores of 0.763 / 0.733 / 0.879 on mammary, thyroid, and liver subsets, respectively. The corresponding clinical F1 scores were 0.905 / 0.918 / 0.900. For image-to-text retrieval, the top-1 relevance score improved from 0.27 to 0.71, while the bottom-1 mismatched score fell from 0.22 to 0.05.

The gains are not just about longer text: the clinical precision, recall, and F1 results suggest that the generated reports remain better aligned with the relevant findings.

Main experiment results

Split	Method	B-1	B-2	B-3	B-4	METEOR	ROUGE-L	Accuracy	Precision	Recall	F1
Mammary	CNN-RNN	0.114	0.093	0.078	0.067	0.221	0.185	0.000	0.496	0.498	0.487
	TriNet	0.693	0.594	0.533	0.478	0.439	0.742	0.351	0.816	0.697	0.727
	R2Gen	0.663	0.611	0.572	0.541	0.411	0.685	0.494	0.800	0.761	0.776
	Transformer	0.699	0.653	0.619	0.590	0.437	0.757	0.461	0.827	0.671	0.702
	DeltaNet	0.716	0.665	0.638	0.608	0.517	0.758	0.573	0.819	0.819	0.818
	R2GenRL	0.672	0.595	0.531	0.479	0.500	0.651	0.424	0.793	0.754	0.771
	SGF	0.761	0.710	0.672	0.640	0.468	0.758	0.586	0.815	0.831	0.822
	Ours	0.763	0.711	0.670	0.637	0.470	0.755	0.547	0.905	0.906	0.905
Thyroid	CNN-RNN	0.131	0.105	0.086	0.069	0.069	0.207	0.000	0.448	0.348	0.382
	TriNet	0.645	0.510	0.421	0.345	0.409	0.678	0.268	0.845	0.769	0.803
	R2Gen	0.578	0.532	0.492	0.457	0.369	0.664	0.404	0.810	0.768	0.779
	Transformer	0.709	0.642	0.585	0.538	0.425	0.701	0.260	0.717	0.732	0.724
	DeltaNet	0.610	0.559	0.515	0.579	0.443	0.685	0.363	0.837	0.784	0.795
	R2GenRL	0.616	0.595	0.464	0.414	0.470	0.599	0.434	0.834	0.819	0.826
	SGF	0.729	0.666	0.613	0.568	0.439	0.723	0.524	0.838	0.850	0.841
	Ours	0.733	0.670	0.615	0.568	0.440	0.726	0.514	0.912	0.924	0.918
Liver	CNN-RNN	0.049	0.026	0.011	0.000	0.119	0.102	0.000	0.181	0.068	0.070
	TriNet	0.868	0.821	0.785	0.750	0.531	0.861	0.039	0.898	0.809	0.814
	R2Gen	0.866	0.842	0.822	0.805	0.537	0.869	0.530	0.875	0.880	0.870
	Transformer	0.855	0.832	0.815	0.800	0.524	0.873	0.444	0.749	0.785	0.765
	DeltaNet	0.873	0.846	0.825	0.808	0.593	0.862	0.568	0.900	0.878	0.874
	R2GenRL	0.853	0.818	0.791	0.769	0.575	0.842	0.466	0.885	0.875	0.879
	SGF	0.872	0.848	0.828	0.813	0.539	0.875	0.541	0.879	0.894	0.883
	Ours	0.879	0.851	0.828	0.810	0.544	0.874	0.579	0.901	0.899	0.900

Ablation results

Split	Setting	B-1	B-4	METEOR	ROUGE-L
Mammary	TF	0.699	0.590	0.437	0.757
	w/ Matcher	0.733	0.598	0.455	0.736
	w/ Dice	0.729	0.586	0.446	0.731
	Ours	0.763	0.637	0.470	0.755
	Ours + cls	0.757	0.612	0.460	0.755
Thyroid	TF	0.709	0.538	0.425	0.701
	w/ Matcher	0.728	0.560	0.435	0.723
	w/ Dice	0.690	0.529	0.416	0.715
	Ours	0.733	0.568	0.440	0.726
	Ours + cls	0.729	0.562	0.434	0.726
Liver	TF	0.855	0.800	0.524	0.873
	w/ Matcher	0.879	0.809	0.544	0.866
	w/ Dice	0.879	0.814	0.544	0.871
	Ours	0.879	0.810	0.544	0.874
	Ours + cls	0.879	0.810	0.544	0.871

The extra classification-head experiment was not consistently helpful: Ours + cls slightly decreases the Mammary and Thyroid scores, while Liver remains nearly unchanged.

Case study

Example of a generated ultrasound report — An example comparison between the reference report, a vanilla Transformer, and CURDNet.

Conclusion

Diversity-aware sampling and cross-modal matching complement each other: one changes what the model sees, while the other changes how it checks image-text alignment. The main limitation is that EchoDice uses organ labels, so its generalization to unlabeled or unseen organs is still an open question. The study is also a research prototype, not a clinical decision-making system.

Links

More details regarding this project can be found on: