Top 5 Image Captioning Datasets Every Computer Vision Researcher Must Know

A Complete Guide to Image-Caption Paired Datasets, Annotation Styles, Metrics, Models, and Implementation — With Final Year Project Angles for Researchers, PhD, M.Tech, and Final Year Students

Introduction
What is Image Captioning?
Why Datasets Matter More Than You Think
How This Article is Structured
What Makes a Great Image Captioning Dataset?
Dataset 1 — MS COCO Captions
Dataset 2 — Flickr30k
Dataset 3 — Visual Genome
Dataset 4 — Conceptual Captions (CC3M / CC12M)
Dataset 5 — NoCaps
Image Captioning Metrics Explained
Comparison Table — All 5 Datasets Across 12+ Attributes
How to Choose the Right Dataset
Common Captioning Models Benchmarked
How to Prepare Image-Caption Pairs for Training
Research Gap Radar — 5 Open Problems
Implementation Roadmap — 8-Step Week-by-Week Guide
Tools and Frameworks
7 Common Mistakes Researchers Make
Your Next Steps + Conclusion

Introduction

What is Image Captioning?

Image captioning is the task of generating a natural-language description y for an input image x, i.e. learning a mapping x → y where y is a grammatically correct sentence describing the salient content, relationships, and context of the image. It sits at the intersection of computer vision and natural language processing, and is one of the clearest testbeds for "grounded" language generation — a model cannot produce a good caption without actually understanding what is in the image.

Captioning research has moved through three broad eras:

Template and retrieval era (2000s–early 2010s): Captions were generated by detecting objects/attributes and slotting them into hand-written templates, or by retrieving the closest matching caption from a database of human-written sentences.
Encoder-decoder era (2014–2019): CNN encoders (VGG, ResNet) paired with RNN/LSTM decoders, often with attention (Show, Attend and Tell), learned to generate captions end-to-end on COCO and Flickr30k.
Vision-language pretraining era (2020–present): Large-scale web-scraped datasets (Conceptual Captions, LAION) combined with transformer encoder-decoders (BLIP, GIT, Flamingo, LLaVA) enabled captioning models that generalise far beyond the narrow domains of COCO and Flickr.

Why Datasets Matter More Than You Think

A model's reported BLEU/CIDEr score is meaningless without knowing which dataset it was tested on, and what kind of captions that dataset contains. Here is why this matters for you:

Annotation style drives output style. COCO captions are short, literal, single-sentence descriptions. Visual Genome region captions are dense and localised. A model fine-tuned only on COCO will sound "flat" compared to one exposed to Conceptual Captions' web-style phrasing.
Reproducibility depends on standardised splits. The "Karpathy split" of COCO (113,287 train / 5,000 val / 5,000 test images) is the de facto standard for comparability across papers. Using a random split is a red flag to reviewers.
Dataset licensing constrains deployment. COCO and Flickr30k images carry their own licences (mostly research-friendly but not unrestricted). Conceptual Captions is built from scraped alt-text, which has its own copyright considerations for the underlying images.
Compute budget is gated by dataset size. COCO has ~120K images with 5 captions each (~600K captions). Conceptual Captions has 3M–12M image-text pairs. Knowing which variant to use can save weeks of preprocessing and training time.

How This Article is Structured

Each of the five dataset sections follows the same 12-subsection template: overview, origin, annotation characteristics, image/caption statistics, download/access, licence, how researchers use it, code to load it, reported state-of-the-art numbers, known limitations, research angles, and a quick-reference summary card. After the datasets, you get metrics, model benchmarks, data preparation recipes, a research gap radar, a week-by-week implementation roadmap, and the tooling ecosystem.

Read end to end for the first time. Return to individual sections as a reference.

What Makes a Great Image Captioning Dataset?

1. Caption Diversity and Style

A great dataset doesn't just have one correct caption per image — it has multiple human-written captions per image (typically 5), capturing different valid descriptions of the same scene. This diversity is essential for metrics like CIDEr and SPICE, which compare against a set of references, and for training models that don't collapse onto a single generic phrasing ("a person standing in a room").

2. Coverage of Objects, Attributes, and Relationships

Captions that go beyond "what objects are present" to describe attributes (colour, size, material) and relationships (spatial, action-based) push models toward genuine scene understanding rather than object-list regurgitation. Visual Genome's relationship triplets are the clearest example of this.

3. Image Diversity

Diversity means: variety of scene types (indoor, outdoor, urban, natural), variety of subjects (people, animals, objects, abstract scenes), variety of image sources (professional photography, smartphone snapshots, web images), and variety of resolutions and aspect ratios. Datasets sourced entirely from one platform (e.g. only Flickr) inherit that platform's photographic biases.

4. Annotation Quality and Consistency

Crowd-sourced captions (COCO, Flickr30k) go through quality-control pipelines (e.g. rejecting captions under a certain length, filtering for profanity, checking for copy-pasted text). Web-scraped alt-text datasets (Conceptual Captions) trade quality control for scale — alt-text is noisy, sometimes irrelevant, and requires aggressive filtering.

5. Licence and Accessibility

Some datasets distribute images directly; others (like Conceptual Captions) distribute only URLs, meaning a non-trivial fraction of images become unavailable over time as links rot. Before building a paper, repo, or product around a dataset, check both the licence of the captions and the underlying images.

Dataset 1 — MS COCO Captions

1.1 Overview

MS COCO Captions is the single most important benchmark in image captioning. Built on top of the Microsoft Common Objects in Context (COCO) detection/segmentation dataset, it adds five human-written captions per image for ~123,000 images. If you have read any captioning paper published between 2015 and 2024, you have almost certainly seen a results table reporting BLEU, METEOR, ROUGE-L, CIDEr, and SPICE on the COCO Karpathy test split.

1.2 Origin and History

COCO was introduced by Lin et al. (2014, ECCV) as a large-scale object detection and segmentation dataset, collected by querying Flickr for everyday scenes containing common objects in natural context (hence "Common Objects in Context"). Captions were added via Amazon Mechanical Turk, with five independent annotators per image instructed to write a single descriptive sentence without naming specific people or making subjective judgements.

The "Karpathy split" — introduced by Karpathy and Fei-Fei in their 2015 Neural Talk paper — repartitions the official train/val sets into 113,287 training images, 5,000 validation images, and 5,000 test images, and has become the universal evaluation protocol.

1.3 Annotation Characteristics

COCO captions follow a consistent style: short (typically 8–12 words), literal, single-sentence, present-tense descriptions of the main subject and action ("A man riding a skateboard down a ramp"). Annotators were explicitly instructed not to describe things that might happen in the future, not to give opinions, and not to include text visible in the image.

Implications:

High inter-annotator consistency makes COCO ideal for supervised training with cross-entropy / reinforcement learning (CIDEr-optimisation) objectives.
The narrow style means models trained purely on COCO produce somewhat generic, "templated" captions when applied to out-of-domain images.

1.4 Image Statistics

Attribute	Value
Number of images	~123,287 (train + val, Karpathy split)
Captions per image	5
Total captions	~616,000
Average caption length	~10.5 words
Vocabulary size	~27,000 words (full), ~10,000 (frequency-filtered)
Image source	Flickr
Image content	Everyday scenes, 80 object categories

1.5 Download and Access

Official COCO site: https://cocodataset.org/#download — images and 2014/2017 caption annotations.
Karpathy split JSON (widely used): hosted via Andrej Karpathy's neuraltalk2 repository and mirrored in most captioning codebases (e.g. dataset_coco.json).
Hugging Face mirror: search "coco captions" on huggingface.co/datasets for a ready-to-stream version.

1.6 Licence

Images are sourced from Flickr under a variety of Creative Commons licences (mostly CC BY 2.0 / CC BY-NC 2.0), and annotations are released by Microsoft under a CC BY 4.0 licence for research use. Verify the per-image licence before any commercial deployment, as Flickr source licences vary.

1.7 How Researchers Use COCO Captions

The standard pipeline is:

Train on the 113,287-image Karpathy training split.
Validate hyperparameters on the 5,000-image validation split.
Report BLEU-1 through BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE on the 5,000-image test split.
Optionally, submit to the official COCO Captioning Challenge leaderboard, which evaluates on a separate hidden test set with 40 reference captions per image (for fairer n-gram metric computation).

Models are almost always pretrained on COCO before being evaluated for zero-shot or fine-tuned transfer to Flickr30k, NoCaps, or other datasets — making COCO the "ImageNet of captioning."

1.8 Code to Load and Evaluate on COCO Captions

import json
from pycocotools.coco import COCO
from pycocoevalcap.eval import COCOEvalCap

def load_coco_captions(ann_file):
    """Load COCO caption annotations using pycocotools."""
    coco = COCO(ann_file)
    img_ids = coco.getImgIds()
    print(f"Loaded {len(img_ids)} images with caption annotations")
    return coco, img_ids

def evaluate_coco_captions(gt_ann_file, result_file):
    """
    Evaluate generated captions against COCO ground truth.
    result_file: JSON list of {"image_id": int, "caption": str}
    """
    coco = COCO(gt_ann_file)
    coco_res = coco.loadRes(result_file)

    coco_eval = COCOEvalCap(coco, coco_res)
    coco_eval.params['image_id'] = coco_res.getImgIds()
    coco_eval.evaluate()

    for metric, score in coco_eval.eval.items():
        print(f"{metric}: {score:.4f}")
    return coco_eval.eval

1.9 State-of-the-Art Numbers on COCO Karpathy Test Split

Model	Year	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
Show & Tell	2015	27.7	23.7	—	85.5	—
Show, Attend & Tell	2015	24.3	23.9	—	—	—
Bottom-Up Top-Down	2018	36.3	27.7	56.9	120.1	21.4
Oscar	2020	37.4	30.7	60.8	127.8	23.5
BLIP	2022	40.4	—	—	136.7	—
BLIP-2 (OPT-2.7B)	2023	43.7	—	—	145.8	—
GIT2	2022	—	—	—	148.6	—

Scores reported on the Karpathy test split with CIDEr-optimisation where applicable. Numbers sourced from respective papers; minor differences exist across beam-search settings.

1.10 Known Limitations

Narrow caption style. Short, literal captions trained models toward generic phrasing; rare objects and complex relationships are under-described.
Object-category bias. Images are selected to contain instances of 80 predefined COCO object categories, which biases the visual distribution toward "common objects."
Five captions is not exhaustive. Many valid descriptions of an image are absent, which penalises models producing equally valid but unseen phrasings under n-gram metrics.
Western/English-centric. Annotations are English-only and reflect annotator cultural assumptions about what is "worth describing."

1.11 Research Angles for Final Year / PhD Students

Caption diversity metrics: Propose a metric that rewards lexical/semantic diversity across the 5 reference captions rather than just n-gram overlap with the top prediction.
Controllable captioning: Build a model that can generate captions emphasising different objects/regions on request, evaluated via COCO region annotations.
Bias auditing: Analyse COCO captions for gender, occupation, and activity stereotypes (e.g., are kitchens disproportionately captioned with female pronouns?) and propose a debiasing fine-tuning recipe.
Cross-lingual COCO: Use machine-translated COCO captions (e.g. COCO-CN, STAIR Captions for Japanese) to study cross-lingual transfer.

1.12 Quick Reference Card

Dataset 2 — Flickr30k

2.1 Overview

Flickr30k is the second pillar of classical image captioning benchmarks, alongside COCO. Introduced by Young et al. (2014), it consists of 31,783 images collected from Flickr, each annotated with five crowd-sourced captions. Flickr30k predates COCO Captions and remains widely used as a secondary evaluation set and for the closely related task of image-text retrieval, especially via its extended annotation set, Flickr30k Entities.

2.2 Origin and History

Flickr30k extends the earlier Flickr8k dataset (Hodosh et al., 2013) from 8,000 to ~31,800 images, all sourced from Flickr and depicting people engaged in everyday activities. Captions were collected via Amazon Mechanical Turk using a similar protocol to COCO (five independent annotators per image).

A major extension, Flickr30k Entities (Plummer et al., 2015), adds bounding-box annotations linking noun phrases in each caption to specific regions in the image — making it a key resource for grounded/referring captioning and phrase localisation research.

2.3 Annotation Characteristics

Flickr30k captions are similar in style to COCO but tend to be slightly longer and more descriptive of human activity, since the source images were curated to feature people in everyday scenarios ("Two young, White males are outside near many bushes," "Several men in hard hats are operating a giant pulley system"). The Flickr30k Entities extension adds:

Noun phrase chunking for each of the 5 captions per image.
Bounding boxes linking each noun phrase to image regions (where the entity is visible).
Coreference chains across the 5 captions for the same image.

2.4 Image Statistics

Attribute	Value
Number of images	31,783
Captions per image	5
Total captions	~158,915
Average caption length	~13 words
Standard split (Karpathy)	29,000 train / 1,014 val / 1,000 test
Flickr30k Entities	+275,000 bounding boxes, ~513,000 noun phrases

2.5 Download and Access

Official request form: https://shannon.cs.illinois.edu/DenotationGraph/ — original Flickr30k release page.
Flickr30k Entities: https://github.com/BryanPlummer/flickr30k_entities — bounding box and phrase annotations.
Karpathy split JSON: same neuraltalk2-style repositories that host the COCO split also host dataset_flickr30k.json.
Hugging Face mirror: search "flickr30k" on huggingface.co/datasets.

2.6 Licence

Images remain the property of their original Flickr uploaders under their individual Creative Commons licences. The caption annotations themselves are released for academic research use; redistribution of the raw images is generally handled via a request-and-agreement process with the original dataset authors.

2.7 How Researchers Use Flickr30k

Two standard usage modes:

Secondary captioning benchmark: Train on COCO, evaluate zero-shot or fine-tune-then-evaluate on Flickr30k's 1,000-image Karpathy test split, to demonstrate that results aren't overfit to COCO's specific style.
Image-text retrieval: Flickr30k (1K test set) is one of the two standard retrieval benchmarks (alongside COCO 5K) for evaluating CLIP-style dual encoders on Recall@1/5/10 for both image-to-text and text-to-image retrieval.
Phrase grounding: Flickr30k Entities is the standard benchmark for evaluating whether a model's attention or grounding correctly localises the noun phrases mentioned in a caption.

2.8 Code to Load Flickr30k Captions and Entities

import json
import os
from xml.etree import ElementTree as ET

def load_flickr30k_karpathy(json_path):
    """Load Flickr30k Karpathy-split captions."""
    with open(json_path, 'r') as f:
        data = json.load(f)

    splits = {'train': [], 'val': [], 'test': []}
    for img in data['images']:
        split = img['split']
        captions = [s['raw'] for s in img['sentences']]
        splits[split].append({
            'filename': img['filename'],
            'captions': captions
        })
    for split, items in splits.items():
        print(f"{split}: {len(items)} images")
    return splits

def load_flickr30k_entities_annotation(xml_path):
    """Parse a single Flickr30k Entities annotation XML file for bounding boxes."""
    tree = ET.parse(xml_path)
    root = tree.getroot()

    boxes = []
    for obj in root.findall('.//object'):
        name = obj.find('name').text
        bndbox = obj.find('bndbox')
        if bndbox is not None:
            box = {
                'entity_id': name,
                'xmin': int(bndbox.find('xmin').text),
                'ymin': int(bndbox.find('ymin').text),
                'xmax': int(bndbox.find('xmax').text),
                'ymax': int(bndbox.find('ymax').text),
            }
            boxes.append(box)
    return boxes

2.9 State-of-the-Art Numbers on Flickr30k

Model	Year	BLEU-4	METEOR	CIDEr	Retrieval R@1 (I→T)
Show, Attend & Tell	2015	19.9	18.5	—	—
Bottom-Up Top-Down	2018	27.3	21.7	56.6	—
Oscar	2020	—	—	—	75.0
CLIP (zero-shot retrieval)	2021	—	—	—	88.0
BLIP	2022	30.1	—	—	96.6
BLIP-2	2023	—	—	—	97.6

Captioning numbers (BLEU-4, METEOR, CIDEr) reported on the 1,000-image Karpathy test split. Retrieval R@1 (image-to-text) reported on the standard 1K retrieval test set. Numbers sourced from respective papers.

2.10 Known Limitations

Small test set. 1,000 test images is small; per-model variance can be 1–2 CIDEr points purely from random seed differences.
Human-centric bias. The dataset was deliberately curated to focus on people in activities, limiting coverage of object-centric or scene-centric imagery compared to COCO.
Licence friction. Unlike COCO's direct download, Flickr30k requires a request form, which slows down reproducibility for newcomers.
Demographic descriptors in captions. Some Flickr30k captions include race/gender descriptors (as in the example above) that raise fairness considerations for downstream model behaviour.

2.11 Research Angles for Final Year / PhD Students

Phrase grounding evaluation: Use Flickr30k Entities to evaluate whether attention maps in a transformer captioning model align with ground-truth noun-phrase bounding boxes.
Demographic bias study: Quantify how often Flickr30k captions include race/gender/age descriptors, and measure whether captioning models trained on this data reproduce or amplify these patterns on held-out images.
Cross-dataset transfer: Train on COCO, fine-tune on a small subset (e.g. 1,000 images) of Flickr30k, and study sample-efficient domain adaptation between the two caption styles.
Retrieval-augmented captioning: Use Flickr30k's retrieval annotations to build a retrieval-augmented captioner that conditions generation on similar captions retrieved from a memory bank.

2.12 Quick Reference Card

Flickr30k | 31,783 images, 5 captions each (~159K total) | Human-activity-focused descriptive captions | Research-use, request-based access | Use as: secondary benchmark + retrieval + grounding (via Entities) | Primary metrics: BLEU/METEOR/CIDEr, Recall@K | Download: shannon.cs.illinois.edu/DenotationGraph

Dataset 3 — Visual Genome

3.1 Overview

Visual Genome (VG) is not just a captioning dataset — it is a large-scale scene-understanding dataset that happens to contain one of the richest sources of region-level captions available. Introduced by Krishna et al. (2017), it connects images to structured annotations of objects, attributes, relationships, and region descriptions, making it the go-to resource for dense captioning (generating a caption for every salient region of an image, not just one caption for the whole image).

3.2 Origin and History

Visual Genome was created at Stanford University with the explicit goal of moving beyond "what is in the image" toward "how do the things in the image relate to each other." The dataset reuses ~108,000 images from COCO and YFCC100M, but adds an enormous layer of new annotations: an average of 35 objects, 26 attributes, and 21 pairwise relationships per image, plus ~50 region descriptions per image — short captions describing specific bounding-box regions rather than the image as a whole.

3.3 Annotation Characteristics

Visual Genome annotations come in several interconnected layers:

Region descriptions: ~5.4 million short phrases/sentences, each tied to a bounding box (e.g. "a red fire hydrant on the corner," "two children playing with a ball").
Relationship triplets: subject–predicate–object structures (e.g. "man – riding – horse"), forming the basis of scene graphs.
Object and attribute annotations: dense labelling of objects with associated attributes (colour, size, material, state).
Question-Answer pairs: ~1.7 million QA pairs, used for visual question answering research.

For dense captioning, region descriptions are the primary annotation of interest — they are shorter and more localised than whole-image COCO captions, often describing a single object plus its immediate context.

3.4 Image Statistics

Attribute	Value
Number of images	~108,000
Region descriptions	~5.4 million (avg ~50/image)
Relationship triplets	~2.3 million
Objects annotated	~3.8 million
Attributes annotated	~2.8 million
Image source	COCO + YFCC100M (Flickr)
Average region caption length	~5–7 words

3.5 Download and Access

Official site: https://homes.cs.washington.edu/~ranjay/visualgenome/index.html — images, region descriptions, scene graphs, and QA pairs, split across two versions (v1.0, v1.4).
Region descriptions JSON: region_descriptions.json available directly from the official downloads page.
Hugging Face mirror: search "visual genome" on huggingface.co/datasets for a preprocessed version with region-caption pairs.

3.6 Licence

Visual Genome annotations are released under a Creative Commons Attribution 4.0 licence. Since many images are reused from COCO (itself sourced from Flickr under various CC licences), the same per-image licence caveats as COCO apply to the underlying imagery.

3.7 How Researchers Use Visual Genome

Visual Genome supports several distinct research directions, and how it's used depends heavily on which annotation layer is targeted:

Dense captioning: Train a model (e.g. DenseCap-style) to jointly localise regions and generate a caption for each, evaluated with mean Average Precision (mAP) combining localisation IoU and caption quality (METEOR).
Scene graph generation: Predict the full graph of objects, attributes, and relationships for an image — a structured precursor to caption generation.
Pretraining corpus: Region descriptions are commonly mixed into large-scale vision-language pretraining corpora (used by models like Oscar, VinVL) because of their density and grounding to specific image regions.
Visual grounding / referring expressions: Region descriptions paired with bounding boxes serve as a source of referring-expression data.

3.8 Code to Load Visual Genome Region Descriptions

import json
from collections import defaultdict

def load_vg_region_descriptions(region_json_path, max_images=None):
    """
    Load Visual Genome region descriptions.
    Returns a dict: image_id -> list of {phrase, x, y, width, height}
    """
    with open(region_json_path, 'r') as f:
        data = json.load(f)

    image_regions = {}
    for i, entry in enumerate(data):
        if max_images and i >= max_images:
            break
        image_id = entry['id']
        regions = []
        for region in entry['regions']:
            regions.append({
                'phrase': region['phrase'],
                'x': region['x'],
                'y': region['y'],
                'width': region['width'],
                'height': region['height'],
            })
        image_regions[image_id] = regions

    total_regions = sum(len(r) for r in image_regions.values())
    print(f"Loaded {len(image_regions)} images, {total_regions} region descriptions")
    return image_regions

3.9 State-of-the-Art Numbers on Visual Genome (Dense Captioning, mAP)

Model	Year	mAP (IoU 0.3–0.7)	METEOR (region-level)
DenseCap (FCLN)	2016	5.39	11.0
JIVC (Joint Inference)	2017	9.31	12.7
CAG-Net	2019	10.51	13.9
TDC (Transformer Dense Cap)	2021	11.49	14.5

mAP combines region localisation accuracy (IoU thresholds 0.3–0.7) with caption quality (METEOR ≥ a threshold). Numbers vary considerably across papers due to differing region-proposal pipelines.

3.10 Known Limitations

Annotation noise. With ~5.4 million crowd-sourced region descriptions, quality is inconsistent — some regions are mislabelled, overlapping, or trivially short ("a chair").
Heavy overlap with COCO. Since many images are shared with COCO, care must be taken to avoid train/test leakage when combining VG pretraining with COCO caption evaluation.
Region descriptions are not full sentences. Many are noun phrases rather than complete sentences, which complicates direct comparison with whole-image captioning metrics.
Two incompatible versions. v1.0 and v1.4 have different object/region counts and category vocabularies, and many papers do not specify which version they used.

3.11 Research Angles for Final Year / PhD Students

Scene-graph-conditioned captioning: Use VG's relationship triplets as an intermediate representation — predict a scene graph first, then generate a caption conditioned on the graph, and measure whether this improves relational accuracy over direct image-to-text models.
Dense captioning efficiency: Propose a lightweight dense captioning head that can run on top of an existing object detector (e.g. YOLO) without a separate region-proposal network.
Noise-robust training: Develop a filtering or re-weighting scheme for VG's noisy region descriptions and measure downstream improvement on dense captioning mAP.
VG as pretraining data: Ablate the contribution of VG region descriptions vs. COCO captions in a vision-language pretraining mix, measuring downstream performance on COCO and NoCaps.

3.12 Quick Reference Card

Visual Genome | ~108K images, ~5.4M region descriptions + scene graphs + QA pairs | Localised region-level phrases/captions | CC BY 4.0 (annotations) | Use as: dense captioning + scene graphs + pretraining corpus | Primary metrics: mAP (localisation + METEOR), Recall@K (graphs) | Download: homes.cs.washington.edu/~ranjay/visualgenome

Dataset 4 — Conceptual Captions (CC3M / CC12M)

4.1 Overview

Conceptual Captions is the dataset that bridges classical captioning benchmarks and modern large-scale vision-language pretraining. Introduced by Google Research (Sharma et al., 2018) as CC3M (3.3 million image-caption pairs) and later extended to CC12M (12 million pairs), it is built by automatically harvesting and cleaning image alt-text from across the web — making it orders of magnitude larger than COCO or Flickr30k, at the cost of caption quality and consistency.

4.2 Origin and History

Conceptual Captions was built via an automatic pipeline that crawls billions of web pages, extracts <img> tags with associated alt-text, and applies a sequence of filters: image-based filters (removing pornographic, low-quality, or non-photographic images), text-based filters (removing alt-text that is too short, too long, contains rare proper nouns, or fails NLP well-formedness checks), and a hypernymisation step that replaces specific proper nouns (e.g. "Harrison Ford") with general concepts (e.g. "actor") to protect privacy and improve generalisation.

CC12M (2021) relaxes several of these filters to roughly quadruple the dataset size, prioritising scale and diversity of visual concepts over the stricter cleanliness of CC3M — explicitly targeting better coverage of long-tail visual concepts for vision-language pretraining.

4.3 Annotation Characteristics

Unlike COCO and Flickr30k, Conceptual Captions has one caption per image (the cleaned alt-text), not five. Captions are typically short, often noun-phrase-like, and reflect how people actually describe images on the web — including marketing language, stock-photo descriptions, and news captions ("a man wearing a hard hat speaks to reporters during a press conference about a new construction project").

Key characteristics:

Captions are weakly aligned — alt-text doesn't always describe everything visible, and sometimes describes context not visible in the image at all (e.g. "actor pictured at last week's premiere").
Hypernymisation (CC3M) means specific named entities are replaced with generic categories, reducing the dataset's usefulness for entity-specific captioning but improving generalisation.
No human-verified multi-caption sets — each image has exactly one (noisy) caption.

4.4 Image Statistics

Attribute	Value
CC3M pairs	~3.3 million
CC12M pairs	~12 million
Captions per image	1
Average caption length	~10 words
Image source	Open web (alt-text harvesting)
Distribution format	URL + caption text (images must be downloaded)

4.5 Download and Access

Official CC3M page: https://ai.google.com/research/ConceptualCaptions/ — TSV files of (URL, caption) pairs for train and validation splits.
CC12M page: https://github.com/google-research-datasets/conceptual-12m — TSV file of (URL, caption) pairs.
Hugging Face mirror: search "conceptual captions" on huggingface.co/datasets — some mirrors include pre-downloaded image subsets, since URL rot affects 10–30% of links over time.
Image downloading tool: the img2dataset package is the community-standard tool for efficiently downloading and resizing millions of images from URL lists.

# Example: download CC3M using img2dataset
pip install img2dataset
img2dataset --url_list cc3m.tsv --input_format "tsv" \
  --url_col "url" --caption_col "caption" \
  --output_folder cc3m_images --processes_count 16 --thread_count 64 \
  --resize_mode "keep_ratio" --image_size 256

4.6 Licence

The caption text and URL list are released by Google for academic and research use. Critically, the images themselves are not redistributed — they remain under whatever licence their original web hosts apply, and researchers download them individually. This means a Conceptual Captions–trained model's underlying images may have heterogeneous and largely unverifiable licensing.

4.7 How Researchers Use Conceptual Captions

Conceptual Captions is used almost exclusively as a pretraining corpus, not as a final evaluation benchmark:

Pretrain a vision-language model (contrastive, generative, or both) on CC3M/CC12M, often combined with other large corpora (SBU Captions, LAION subsets, Visual Genome region descriptions).
Fine-tune the pretrained model on COCO Captions for standard captioning evaluation, or on COCO/Flickr30k for retrieval.
Evaluate zero-shot transfer on NoCaps to measure how well the pretraining corpus's broader concept coverage translates into captioning novel objects not seen in COCO.

The core research question CC3M/CC12M enables is: how much does pretraining on noisy, large-scale, diverse web data improve generalisation beyond what COCO alone can teach?

4.8 Code to Load and Filter Conceptual Captions

import csv
import os
from img2dataset import download

def load_cc_tsv(tsv_path, max_rows=None):
    """Load Conceptual Captions TSV (caption, url) pairs."""
    pairs = []
    with open(tsv_path, 'r', encoding='utf-8') as f:
        reader = csv.reader(f, delimiter='\t')
        for i, row in enumerate(reader):
            if max_rows and i >= max_rows:
                break
            caption, url = row[0], row[1]
            pairs.append({'caption': caption, 'url': url})
    print(f"Loaded {len(pairs)} caption-URL pairs")
    return pairs

def download_cc_images(tsv_path, output_dir, image_size=256, processes=8):
    """Download Conceptual Captions images using img2dataset."""
    download(
        url_list=tsv_path,
        output_folder=output_dir,
        input_format="tsv",
        url_col="url",
        caption_col="caption",
        output_format="files",
        image_size=image_size,
        resize_mode="keep_ratio",
        processes_count=processes,
        thread_count=64,
    )
    print(f"Downloaded images to {output_dir}")

4.9 State-of-the-Art Numbers (Effect of CC Pretraining on Downstream Captioning)

Pretraining Corpus	Fine-tune Target	Year	CIDEr (COCO)	CIDEr (NoCaps overall)
COCO only	COCO	2018	120.1	61.5
CC3M + COCO	COCO	2020	127.8	78.3
CC12M + VG + COCO	COCO	2021	130.4	92.5
CC12M + LAION subset + COCO	COCO	2022	136.7	105.7

Illustrative numbers compiled from comparative studies; exact values depend heavily on model architecture and additional data mixed in. The key trend — larger/noisier pretraining corpora substantially improve NoCaps performance more than COCO performance — is consistent across papers.

4.10 Known Limitations

URL rot. A meaningful fraction of CC3M/CC12M image URLs become dead over time (estimates range from 10–30%), meaning exact dataset reproduction is impossible; researchers report whatever subset they successfully downloaded.
Single noisy caption per image. No multi-reference evaluation is possible directly on CC3M/CC12M; it is unsuitable as a final benchmark.
Hypernymisation loses specificity (CC3M). Replacing named entities with generic terms can remove information that is actually visually present and useful.
Alt-text ≠ description. A non-trivial fraction of alt-text describes context, intent, or metadata rather than visual content ("Image used under licence from Getty").

4.11 Research Angles for Final Year / PhD Students

Caption quality filtering: Design an automatic scorer (e.g. CLIP-similarity-based) to filter CC12M down to a high-quality subset, and measure whether training on the filtered subset matches or beats training on the full noisy set.
Data scaling laws for captioning: Train identical architectures on CC3M, CC12M, and combinations with COCO/VG at varying data fractions, and fit a scaling curve for CIDEr on NoCaps vs. pretraining data size.
Synthetic caption augmentation: Use a strong captioning model (e.g. BLIP) to re-caption a subset of CC12M images, and compare training on original alt-text vs. synthetic captions vs. a mixture (the "BLIP CapFilt" approach).
Bias in web-scraped captions: Analyse CC3M/CC12M for representation biases (geographic, occupational, gendered) relative to COCO, given that web alt-text reflects global internet content rather than a curated annotation protocol.

4.12 Quick Reference Card

Conceptual Captions (CC3M/CC12M) | 3.3M / 12M image-caption pairs | Web alt-text, 1 caption/image, noisy | Research-use licence (captions/URLs only; images unrestricted-host) | Use as: large-scale pretraining corpus | Primary metrics: downstream CIDEr/SPICE after fine-tuning | Download: ai.google.com/research/ConceptualCaptions

Dataset 5 — NoCaps

5.1 Overview

NoCaps (Novel Object Captioning at Scale) is not a training dataset — it is a test-only benchmark specifically designed to measure how well captioning models generalise to visual concepts they were not trained on. Introduced by Agrawal et al. (2019), NoCaps occupies the same role for captioning that DND occupies for denoising: a held-out, server-evaluated stress test that exposes overfitting to the training distribution (COCO).

5.2 Origin and History

NoCaps was created by researchers to address a specific gap: models trained on COCO achieve impressive CIDEr scores on COCO's test set but were known anecdotally to fail badly when shown images containing objects outside COCO's 80 categories (e.g. "accordion," "hippopotamus," "umbrella stand"). NoCaps formalises this by sourcing images from Open Images — which covers ~600 object classes, far broader than COCO's 80 — and splitting the test set into three tiers based on how many novel (non-COCO) object classes each image contains.

5.3 Annotation Characteristics

NoCaps images each have 11 human-written reference captions (more than COCO's 5), collected via the same general annotation protocol as COCO to maximise comparability of n-gram metrics. The dataset is split into three subsets:

In-Domain: images containing only COCO object classes.
Near-Domain: images containing a mix of COCO and novel (non-COCO) object classes.
Out-of-Domain: images containing only novel object classes never seen in COCO's category list.

Models are typically trained exclusively on COCO (no NoCaps training data exists) and evaluated across all three subsets — the gap between In-Domain and Out-of-Domain CIDEr is the key generalisation signal.

5.4 Image Statistics

Attribute	Value
Validation images	4,500
Test images	10,600
References per image	11
Object vocabulary	~600 classes (from Open Images)
Subsets	In-Domain / Near-Domain / Out-of-Domain
Training data	None — evaluation only

5.5 Download and Access

Official site: https://nocaps.org/ — dataset description, download links, and the evaluation server.
Images: sourced from the Open Images dataset (https://storage.googleapis.com/openimages/web/index.html); NoCaps provides the relevant image ID subsets and annotation files.
Evaluation server: https://eval.ai/web/challenges/challenge-page/355/overview — submit generated captions for the val/test splits to receive official CIDEr/SPICE scores, broken down by In/Near/Out-of-Domain.

5.6 Licence

NoCaps annotations are released for research use. Images are sourced from Open Images, which is distributed under a CC BY 4.0 licence — generally more permissive than COCO or Flickr30k's mixed per-image licensing.

5.7 How Researchers Use NoCaps

NoCaps is a zero-shot generalisation benchmark, used as follows:

Train a captioning model on COCO Captions (and optionally large-scale pretraining corpora like CC12M, SBU, VG).
Run inference on NoCaps val/test images without any NoCaps-specific fine-tuning.
Submit predictions to the EvalAI server, which returns CIDEr and SPICE broken down by In-Domain, Near-Domain, Out-of-Domain, and Overall.
Report the Out-of-Domain CIDEr and the In-Domain to Out-of-Domain gap as the key headline numbers — a small gap indicates strong generalisation.

Papers introducing new large-scale pretraining corpora (CC12M, LAION-based pretraining) almost always report NoCaps numbers as their primary evidence of improved generalisation, since COCO-only numbers tend to saturate.

5.8 Code to Prepare a NoCaps Submission

import json

def prepare_nocaps_submission(image_ids, generated_captions, output_path):
    """
    Prepare a NoCaps submission file in the required EvalAI format.
    image_ids: list of NoCaps image IDs
    generated_captions: list of generated caption strings (same order)
    """
    assert len(image_ids) == len(generated_captions)

    submission = []
    for img_id, caption in zip(image_ids, generated_captions):
        submission.append({
            "image_id": img_id,
            "caption": caption.strip()
        })

    with open(output_path, 'w') as f:
        json.dump(submission, f)

    print(f"Saved {len(submission)} predictions to {output_path}")
    print("Upload this file to https://eval.ai/web/challenges/challenge-page/355/overview")

5.9 State-of-the-Art Numbers on NoCaps (CIDEr, val split)

Model	Year	In-Domain	Near-Domain	Out-of-Domain	Overall
Up-Down (COCO only)	2019	78.1	57.7	31.3	55.3
Oscar	2020	83.4	81.6	77.6	81.1
VinVL	2021	103.7	95.6	83.8	94.3
BLIP	2022	114.9	112.1	115.3	113.2
GIT2	2022	126.9	125.8	130.6	126.9

CIDEr scores on the NoCaps validation split. Note how the In-Domain/Out-of-Domain gap shrinks dramatically as pretraining corpora grow (Up-Down's 47-point gap vs. GIT2's near-zero gap), reflecting the value of large-scale pretraining for generalisation.

5.10 Known Limitations

No training data. NoCaps cannot be used for fine-tuning at all — any model evaluated on it must be trained elsewhere, which complicates controlled ablations.
Open Images category overlap. Some "novel" categories in Near/Out-of-Domain splits are visually or semantically close to COCO categories (e.g. "kitten" vs. COCO's "cat"), softening the generalisation signal somewhat.
English-only references. Like COCO, all 11 references per image are in English, limiting its use for multilingual generalisation studies.
Static benchmark in a moving field. As foundation models trained on web-scale data increasingly treat "novel object" distinctions as less meaningful, NoCaps' headline numbers are approaching saturation for the largest models.

5.11 Research Angles for Final Year / PhD Students

Fine-grained novel object analysis: Break down NoCaps Out-of-Domain performance by individual Open Images categories to identify which novel objects remain hardest, rather than relying on the aggregate score.
Retrieval-augmented zero-shot captioning: Build a model that retrieves visually similar images with known captions from a large unlabelled corpus at inference time to caption truly novel objects, evaluated on NoCaps Out-of-Domain.
Vocabulary expansion via VQA: Use a VQA model to first identify object names present in an image, then condition a captioning decoder on this vocabulary — measure improvement specifically on NoCaps Out-of-Domain.
Saturation analysis: As an analytical (non-model) project, survey published NoCaps numbers over time and project when/whether the In-Domain/Out-of-Domain gap will reach zero, discussing what that implies for the benchmark's continued usefulness.

5.12 Quick Reference Card

Image Captioning Metrics Explained

BLEU (Bilingual Evaluation Understudy)

BLEU measures n-gram precision between a generated caption and one or more reference captions, with a brevity penalty for short outputs. BLEU-4 (using up to 4-grams) is most commonly reported. It is fast, simple, and was borrowed directly from machine translation. Its main weakness for captioning is that it rewards exact phrase matches and penalises semantically equivalent but differently-worded captions.

Typical range: 20–45 (BLEU-4, as a percentage-style score) for COCO-style benchmarks.

METEOR

METEOR improves on BLEU by incorporating stemming, synonym matching (via WordNet), and a different scoring formula that balances precision and recall (weighted toward recall). It correlates better with human judgement than BLEU at the sentence level, and is reported in nearly every captioning paper alongside BLEU.

ROUGE-L

ROUGE-L is based on the longest common subsequence (LCS) between the generated and reference captions, originally developed for summarisation. It captures sentence-level structure similarity better than n-gram-only metrics, but is less commonly the headline metric in captioning compared to CIDEr.

CIDEr (Consensus-based Image Description Evaluation)

CIDEr is the de facto primary metric for image captioning. It computes TF-IDF weighted n-gram similarity between the candidate caption and the set of reference captions, down-weighting n-grams that are common across the whole dataset (and therefore uninformative) and up-weighting n-grams that are distinctive to the specific image. CIDEr correlates strongly with human consensus judgements and is the metric most papers optimise directly via reinforcement learning (CIDEr-optimisation / SCST).

Typical range: 80–150 for strong models on COCO Karpathy test.

SPICE (Semantic Propositional Image Caption Evaluation)

SPICE parses both candidate and reference captions into scene graphs (objects, attributes, relations) and computes an F-score over matched graph tuples. Unlike the n-gram-based metrics above, SPICE directly measures whether the content of the caption is correct, independent of phrasing — making it the metric most aligned with "did the model actually understand the image."

CLIPScore

CLIPScore computes the cosine similarity between a CLIP image embedding and a CLIP text embedding of the generated caption, requiring no reference captions at all. It is increasingly used for evaluating captions on datasets (like CC12M) where multi-reference ground truth doesn't exist, and correlates well with human judgement for overall plausibility — though it can be gamed by captions that are visually relevant but factually wrong in detail.

Which Metric for Which Task?

Scenario	Recommended Metrics
Standard COCO/Flickr30k benchmark	BLEU-4 + METEOR + CIDEr + SPICE
Reinforcement-learning fine-tuning target	CIDEr
Content/factual accuracy focus	SPICE
Reference-free evaluation (web data, CC12M)	CLIPScore
Dense/region captioning	mAP (localisation + METEOR)
Retrieval-style evaluation	Recall@1/5/10

Comparison Table — All 5 Datasets Across 12+ Attributes

Attribute	COCO Captions	Flickr30k	Visual Genome	Conceptual Captions	NoCaps
Year	2014/2015	2014	2017	2018/2021	2019
# Images	~123K	~31.8K	~108K	3.3M / 12M	4.5K val / 10.6K test
Captions/Image	5	5	~50 region captions	1	11
Caption Style	Short, literal, whole-image	Descriptive, human-activity	Localised region phrases	Web alt-text, noisy	Whole-image, COCO-style
Annotation Source	Crowd-sourced (AMT)	Crowd-sourced (AMT)	Crowd-sourced (AMT)	Automated web harvesting	Crowd-sourced (AMT)
Image Source	Flickr	Flickr	COCO + YFCC100M	Open web	Open Images
Ground Truth	Yes (multi-reference)	Yes (multi-reference)	Yes (region-aligned)	Yes (single, noisy)	Yes (11 references)
Use as Train	Yes (primary)	Limited (small)	Yes (dense cap / pretrain)	Yes (pretraining)	No (test only)
Use as Test	Yes (Karpathy split)	Yes (secondary)	Yes (dense captioning)	No	Yes (server-based)
Licence	CC BY 4.0 / mixed CC	Research, request-based	CC BY 4.0	Research (URLs only)	CC BY 4.0
Best For	Primary benchmark	Retrieval + grounding	Dense captioning, scene graphs	Large-scale pretraining	Generalisation testing
Key Metric	CIDEr	CIDEr / Recall@K	mAP + METEOR	Downstream CIDEr	CIDEr by domain split

How to Choose the Right Dataset

By Task

Standard whole-image captioning (new architecture, controlled comparison): Train and evaluate on COCO Captions (Karpathy split). This is the community standard and ensures comparability with hundreds of published papers.
Image-text retrieval: Use COCO 5K and Flickr30k 1K retrieval test sets, reporting Recall@1/5/10 for both directions.
Dense / region-level captioning: Use Visual Genome region descriptions, evaluated with mAP combining localisation and METEOR.
Large-scale vision-language pretraining: Use Conceptual Captions (CC3M/CC12M), typically combined with Visual Genome and/or SBU Captions, then fine-tune on COCO.
Generalisation / zero-shot evaluation: Use NoCaps to measure how well a COCO-trained (or web-pretrained) model handles novel object categories.

By Domain

General-purpose photography: COCO and Flickr30k cover everyday scenes well.
Fine-grained/rare object coverage: Conceptual Captions and NoCaps (via Open Images) cover a much broader visual vocabulary than COCO's 80 categories.
Scene-graph / relational reasoning: Visual Genome is the only one of the five with explicit relationship annotations.
Multilingual captioning: None of the five is natively multilingual; consider translated extensions (e.g. STAIR Captions for Japanese, COCO-CN for Chinese, Multi30K for Flickr30k in German/French/Czech).

By Compute Budget

GPU-constrained (≤8 GB VRAM, <24h training): Fine-tune a pretrained encoder-decoder (e.g. a small ViT + GPT-2) on COCO Captions only. Evaluate on COCO Karpathy test and Flickr30k test.
Standard research (24 GB VRAM, 2–5 day training): Pretrain on a CC3M subset + Visual Genome region descriptions, fine-tune on COCO, evaluate on COCO + Flickr30k + NoCaps.
Large-scale (multi-GPU, >1 week training): Pretrain on CC12M (+ additional web corpora), fine-tune on COCO, and report full NoCaps breakdown (In/Near/Out-of-Domain) as the headline generalisation result.

Common Captioning Models Benchmarked

Show and Tell (2015, Vinyals et al., CVPR): The first end-to-end CNN-encoder + LSTM-decoder captioning model, directly inspired by sequence-to-sequence machine translation. Established the encoder-decoder paradigm for captioning. COCO CIDEr ~85.5.
Show, Attend and Tell (2015, Xu et al., ICML): Introduced visual attention into the decoder, allowing the model to focus on different image regions at each generation step. The conceptual ancestor of nearly all subsequent attention-based captioning architectures.
Bottom-Up Top-Down Attention (2018, Anderson et al., CVPR): Combines bottom-up object-detector features (Faster R-CNN region proposals) with top-down attention in the decoder. Won the 2017 VQA Challenge and significantly advanced COCO captioning, CIDEr ~120.1.
Oscar (2020, Li et al., ECCV): "Object-Semantics Aligned Pretraining" — uses object tags detected in the image as anchor points to align visual and textual modalities during pretraining on large image-text corpora. Strong performance on both COCO and NoCaps (CIDEr ~127.8 / ~81.1).
VinVL (2021, Zhang et al., CVPR): Improves Oscar's visual representations by training a much stronger object detector on a large, diverse set of datasets, demonstrating that better visual features alone yield large captioning gains (NoCaps overall CIDEr ~94.3).
BLIP (2022, Li et al., ICML): Introduces "CapFilt" — using a captioner and a filter to bootstrap and clean noisy web image-text pairs (like those in CC12M), then jointly pretraining for understanding and generation. COCO CIDEr ~136.7, NoCaps ~113.2.
BLIP-2 (2023, Li et al.): Bridges a frozen image encoder and a frozen large language model via a lightweight "Q-Former," dramatically reducing pretraining cost while leveraging the language capabilities of LLMs. COCO CIDEr ~145.8.
GIT / GIT2 (2022, Wang et al.): "Generative Image-to-text Transformer" — a simple single transformer architecture (image encoder + text decoder, no object detector) trained on a massive 0.8B-pair image-text corpus, achieving the smallest In/Out-of-Domain gap on NoCaps to date (CIDEr ~126.9 overall).

How to Prepare Image-Caption Pairs for Training

Tokenisation and Vocabulary

from transformers import AutoTokenizer

def build_caption_tokenizer(model_name="gpt2", max_length=30):
    """Load a tokenizer suitable for caption generation."""
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    if tokenizer.pad_token is None:
        tokenizer.pad_token = tokenizer.eos_token

    def tokenize_caption(caption):
        return tokenizer(
            caption,
            max_length=max_length,
            padding="max_length",
            truncation=True,
            return_tensors="pt"
        )
    return tokenizer, tokenize_caption

Image Preprocessing

from torchvision import transforms

def get_image_transform(image_size=224, train=True):
    """Standard image preprocessing for captioning encoders (e.g. ViT, ResNet)."""
    if train:
        return transforms.Compose([
            transforms.RandomResizedCrop(image_size, scale=(0.8, 1.0)),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.48145466, 0.4578275, 0.40821073],
                                  std=[0.26862954, 0.26130258, 0.27577711]),
        ])
    else:
        return transforms.Compose([
            transforms.Resize(image_size),
            transforms.CenterCrop(image_size),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.48145466, 0.4578275, 0.40821073],
                                  std=[0.26862954, 0.26130258, 0.27577711]),
        ])

Multi-Reference Handling

When a dataset provides multiple captions per image (COCO, Flickr30k, NoCaps), the standard training approach is to sample one caption per image per epoch (so the model sees varied phrasing across epochs), while evaluation uses all references for metric computation:

import random

def sample_training_caption(captions_list):
    """Randomly sample one caption per image for this training step."""
    return random.choice(captions_list)

def get_all_references(captions_list):
    """Return all references for evaluation (used by CIDEr/SPICE/BLEU)."""
    return captions_list

Filtering Web-Scraped Data (Conceptual Captions style)

import re

def filter_caption(caption, min_words=3, max_words=30):
    """Basic quality filter for web-scraped alt-text captions."""
    words = caption.split()
    if not (min_words <= len(words) <= max_words):
        return None
    # Reject captions that look like filenames or metadata
    if re.search(r'\.(jpg|png|gif|jpeg)$', caption.lower()):
        return None
    if re.search(r'^(img|image|photo)[\s_-]*\d+$', caption.lower()):
        return None
    return caption.strip()

Research Gap Radar — 5 Open Problems

Gap 1 — Hallucination in Generative Captioning. Modern captioning models, especially those built on LLM decoders (BLIP-2, GIT2), can confidently describe objects, attributes, or relationships that are not actually present in the image. Standard metrics like CIDEr do not specifically penalise hallucination, and SPICE only partially captures it. Designing hallucination-specific benchmarks and training objectives (e.g. via contrastive negative examples) remains an open and increasingly urgent problem as these models are deployed in accessibility tools.

Gap 2 — Fine-Grained and Compositional Description. Current models excel at naming the main subject and action but struggle with fine-grained attributes (exact colour shades, brand names, precise spatial relationships among many objects) and compositional generalisation (correctly describing novel combinations of familiar objects, e.g. "a cat wearing a spacesuit"). Visual Genome's relationship annotations are underused for evaluating this specific capability.

Gap 3 — Evaluation Beyond Reference-Based Metrics. BLEU, METEOR, CIDEr, and SPICE all assume a fixed set of "correct" reference captions, but a single image often admits dozens of equally valid descriptions emphasising different aspects. CLIPScore partially addresses this but has its own biases (e.g. favouring captions that match CLIP's training distribution). A metric that better captures the space of acceptable captions, rather than overlap with a small reference set, is an open problem.

Gap 4 — Multilingual and Culturally-Aware Captioning. All five datasets covered here are English-only (or English-derived), and annotation instructions reflect Western/English-speaking annotator assumptions about what is "describable" or "notable" in an image. Building and evaluating on genuinely multilingual, culturally diverse captioning datasets — not just machine-translated English captions — remains underexplored relative to the maturity of English captioning research.

Gap 5 — Dense, Grounded, and Controllable Captioning at Scale. Visual Genome demonstrates the value of region-level dense captioning, but at ~108K images it is small relative to whole-image datasets like CC12M. There is no large-scale (multi-million image) dataset combining dense region captions with the scale of web-harvested data — limiting how far dense/grounded captioning models can be pretrained before fine-tuning on the comparatively small Visual Genome.

Implementation Roadmap — 8-Step Week-by-Week Guide

This roadmap is designed for a Final Year / M.Tech student with 2–3 months before submission and access to a single GPU (8–24 GB).

Week 1 — Environment Setup and Baseline Reproduction. Install PyTorch 2.x and Hugging Face transformers. Load a pretrained image encoder (ViT or ResNet) and a small decoder (GPT-2 or a Transformer decoder). Reproduce a baseline Show-and-Tell-style result on a small subset of COCO. Target: a working training loop that overfits a 100-image subset to near-zero loss — this confirms your pipeline is correct.

Week 2 — Dataset Preparation. Download COCO Captions (Karpathy split) and Flickr30k. Write and verify the data loading pipeline from Section 17, including multi-reference handling for evaluation. Visualise 10 image-caption pairs from each dataset and confirm captions display correctly alongside images.

Week 3 — Baseline Training on COCO. Train your encoder-decoder on the full COCO Karpathy training split using cross-entropy loss. Evaluate BLEU-4, METEOR, CIDEr, and SPICE on the test split using pycocoevalcap. Record baseline numbers — this is your comparison anchor.

Week 4 — Implement Your Proposed Component. This is where your contribution lives. Examples: a region-attention module using Visual Genome bounding boxes, a retrieval-augmentation mechanism, a hallucination-detection auxiliary loss, or a CIDEr-optimisation (SCST) fine-tuning stage. Implement and unit-test in isolation first.

Week 5 — Integrate and Train. Integrate your component into the baseline architecture. Train from scratch or fine-tune from the Week 3 checkpoint. Re-evaluate on COCO. Expect the first run to underperform — this is normal; debug via ablation before assuming the idea doesn't work.

Week 6 — Cross-Dataset Evaluation. Evaluate your COCO-trained model (zero-shot, no fine-tuning) on the Flickr30k test split and, if compute allows, on NoCaps val. Record the In-Domain/Out-of-Domain CIDEr gap on NoCaps as a generalisation signal.

Week 7 — Ablation Studies. Remove or modify your key component and measure the CIDEr/SPICE drop on COCO and the NoCaps gap. A good ablation answers: "what specifically does this component contribute to generalisation, not just in-domain performance?" Run 3–5 variants.

Week 8 — Paper Writing. Structure your paper as: Introduction → Related Work → Method → Experiments → Conclusion. In the experiments section, report results in standard table format: dataset × model × BLEU/METEOR/CIDEr/SPICE, plus the NoCaps domain breakdown if computed. Include qualitative examples — generated captions alongside images — for at least five diverse cases, including at least one failure case with discussion.

Tools and Frameworks

pycocoevalcap: The official COCO captioning evaluation toolkit, implementing BLEU, METEOR, ROUGE-L, CIDEr, and SPICE in a unified interface. Essential for any COCO/Flickr30k/NoCaps evaluation. Repository: https://github.com/tylin/coco-caption
Hugging Face Transformers: Provides pretrained vision encoders (ViT, Swin, BEiT), language decoders (GPT-2, OPT, T5), and ready-to-use vision-language models (BLIP, BLIP-2, GIT) via a unified AutoModel interface. Available via pip install transformers.
img2dataset: The community-standard tool for downloading and resizing large-scale image-URL datasets like Conceptual Captions and LAION efficiently and in parallel. Repository: https://github.com/rom1504/img2dataset
LAVIS: Salesforce's "Library for Vision-Language" — a unified framework supporting BLIP, BLIP-2, ALBEF, and CLIP for both training and evaluation across captioning, retrieval, and VQA tasks. Repository: https://github.com/salesforce/LAVIS
pycocotools: The standard library for loading COCO-format annotations (images, captions, instances), used as the foundation for most COCO/Flickr30k/Visual Genome loading pipelines. Available via pip install pycocotools.
EvalAI: The platform hosting the official NoCaps (and many other) leaderboards. Submissions are uploaded as JSON files and scored server-side against held-out references.

7 Common Mistakes Researchers Make

Mistake 1 — Evaluating on the Wrong Split. COCO has multiple split conventions (the official 2014/2017 splits vs. the Karpathy split). Reporting numbers on the wrong split makes your results incomparable to the vast majority of published baselines, even if your numbers are "correct" for the split you used.

Mistake 2 — Comparing Single-Reference to Multi-Reference Metrics. CIDEr and BLEU scores computed against 5 references (COCO) are not directly comparable to scores computed against 1 reference (a CC12M-style dataset) or 11 references (NoCaps) — more references generally inflate scores. Always state how many references were used.

Mistake 3 — Not Reporting SPICE Alongside CIDEr. A model can achieve a high CIDEr score by producing fluent, generic captions that match common n-gram patterns without describing the specific image content correctly. SPICE catches this; omitting it weakens a paper's claims about "understanding."

Mistake 4 — Beam Search Settings Left Unspecified. Beam width (commonly 3–5) and length-normalisation settings can shift CIDEr by several points. Papers that don't specify beam search parameters make their numbers difficult to reproduce or compare.

Mistake 5 — Ignoring the NoCaps Domain Breakdown. Reporting only "NoCaps Overall CIDEr" hides whether a model's improvement comes from genuine novel-object generalisation (Out-of-Domain) or simply from getting better at COCO-like images (In-Domain). Always report all three subsets.

Mistake 6 — Train/Test Leakage via Visual Genome. Because Visual Genome reuses many COCO images, pretraining on VG and then evaluating on COCO test images that also appear in VG constitutes test-set leakage. Always cross-reference image IDs across datasets before combining them.

Mistake 7 — Treating CLIPScore as Ground Truth. CLIPScore is reference-free and convenient, but it reflects CLIP's own biases and training distribution. A caption can score well on CLIPScore while containing factual errors that a human (or SPICE, against real references) would catch. Use it as a supplementary signal, not a replacement for reference-based evaluation where references exist.

Your Next Steps + Conclusion

A Practical Action Plan

You have now read a complete guide to the five most important image captioning datasets in the field. Here is how to turn that knowledge into research output.

If you are a Final Year B.Tech student: Start with COCO Captions. Download the Karpathy split, set up pycocoevalcap, fine-tune a small pretrained encoder-decoder (e.g. ViT + GPT-2 via Hugging Face), and get your first CIDEr score on the test split. Beating a Show-and-Tell-level CIDEr (~85) is the milestone that tells you your pipeline works.
If you are an M.Tech student with a 6-month timeline: Follow the 8-week roadmap in Section 16, extending into a second 8-week cycle for the cross-dataset (Flickr30k + NoCaps) evaluation and ablation studies. Aim to report a full NoCaps domain breakdown — this server-validated, generalisation-focused result significantly strengthens your thesis.
If you are a PhD student: Your contribution should engage with at least one of the five open problems in the Research Gap Radar. Whether it is hallucination, compositional generalisation, evaluation methodology, multilingual captioning, or scaling dense captioning, pick the gap that aligns with your lab's existing expertise and propose a principled solution validated across at least COCO, Flickr30k, and NoCaps.

What You Should Take Away

Image captioning is a field where the community has converged on a clear evaluation hierarchy: COCO for in-domain performance, Flickr30k for secondary confirmation and retrieval, Visual Genome for dense/relational understanding, Conceptual Captions for pretraining scale, and NoCaps for generalisation. Reporting results across this hierarchy — not just on COCO — is a universally understood signal that reviewers know how to interpret.

At the same time, the most influential captioning papers — Show, Attend and Tell; Bottom-Up Top-Down; BLIP; GIT — succeeded because they introduced ideas (attention, object-grounded features, bootstrapped data cleaning, simplified unified architectures) that changed how the field approaches the connection between vision and language, not because they squeezed out an extra CIDEr point on COCO alone.

The datasets reviewed here will remain relevant for years. The models that meaningfully advance the field will come from researchers who understand what each dataset actually measures — and, increasingly, from those who recognise what none of these datasets measure yet.

Image Processing Projects for Final Year