Import Datasets from Hugging Face

Use HuggingfaceDatasetConfig to specify which dataset, subset, split, and how many rows to pull. Pass it as the source argument to dataset.create().

This example imports 50 rows from the SmolLM-Corpus cosmopedia-v2 subset — a collection of synthetic textbook-style content with prompts, generated text, audience labels, and format tags.

import os
from fi.datasets import Dataset, DatasetConfig, HuggingfaceDatasetConfig
from fi.utils.types import ModelTypes

hf_config = HuggingfaceDatasetConfig(
    name="HuggingFaceTB/smollm-corpus",
    subset="cosmopedia-v2",
    split="train",
    num_rows=50,
)

dataset = Dataset(
    dataset_config=DatasetConfig(
        name="smollm-cosmopedia-import",
        model_type=ModelTypes.GENERATIVE_LLM,
    ),
    fi_api_key=os.environ["FI_API_KEY"],
    fi_secret_key=os.environ["FI_SECRET_KEY"],
)

dataset = dataset.create(source=hf_config)

print(f"Dataset created: {dataset.dataset_config.name}")
print(f"Dataset ID: {dataset.dataset_config.id}")

Expected output:

Dataset created: smollm-cosmopedia-import
Dataset ID: a1b2c3d4-...

Navigate to Dataset in the left sidebar. Your new dataset appears in the list. Click it to browse the imported rows and columns.

The cosmopedia-v2 subset includes columns like prompt, text, audience, format, and token_length — ready for evaluation.

The prompt column contains the generation instruction and text contains the generated output — a natural fit for a completeness evaluation that checks whether the output fully addresses the input.

dataset = dataset.add_evaluation(
    name="completeness-check",
    eval_template="completeness",
    required_keys_to_column_names={
        "input": "prompt",
        "output": "text",
    },
    model="turing_small",
    run=True,
    reason_column=True,
)

print("Evaluation 'completeness-check' started")

Expected output:

Evaluation 'completeness-check' started

Pull the evaluated dataset back as a CSV or a pandas DataFrame.

As CSV:

dataset.download(file_path="smollm_scored.csv")
print("Downloaded scored results to smollm_scored.csv")

As pandas DataFrame:

df = dataset.download(load_to_pandas=True)
print("Columns:", list(df.columns))
print(df.head())

Expected output:

Columns: ['prompt', 'text', 'token_length', 'audience', 'format', 'seed_data', 'completeness-check', 'completeness-check_reason']
   prompt                                               text  ...
0  Write a children's story about...                    ...  ...

dataset.delete()
print("Dataset deleted")