There is no single universal robotics data format yet, but several formats and standards are de facto widely used in robotics research and industry.
Among them
Let’s start our exploration from LeRobot v2 format.
It is used in one of the most widely known LeRobot ML robotics frameworks from Hugging Face. It is reused to some extent by many other frameworks, including those from Physical Intelligence, etc.
Let’s explore it using an example of the ALOHA robotics setup dataset from here.
Note, that originally ALOHA was not created in LeRobot formats, it is more like a raw telemetry log, but here it is converted into LeRobot.
LeRobot v2.0 dataset is a folder with the following structure:
.
├── data
│ └── chunk-000
│ ├── episode_000000.parquet
│ ├── ...
│ └── episode_000049.parquet
├── meta
│ ├── episodes_stats.jsonl
│ ├── episodes.jsonl
│ ├── info.json
│ └── tasks.jsonl
└── videos
└── chunk-000
├── observation.images.cam_high
│ ├── episode_000000.mp4
│ ├── ...
│ └── episode_000049.mp4
├── observation.images.cam_left_wrist
│ ├── episode_000000.mp4
│ ├── ...
│ └── episode_000049.mp4
└── observation.images.cam_right_wrist
├── episode_000000.mp4
├── ...
└── episode_000049.mp4
The structure is mostly self-explanatory and contains three subfolders: data, meta, and videos.
data folder contains robot state and action data in Parquet format.meta folder describes what fields are stored in the data folder. In particular, info.json (via its features field) defines exactly what is stored in the Parquet files.videos folder stores MPEG/MP4 video recordings from each available camera.The meta folder provides a detailed description of which data is stored in the dataset.
For example, info.json contains a description of the dataset features, i.e. what is collected and how it is structured:
As a side note, let me emphasize that the term episodes here is closely related to the concept of episodic memory discussed in LeCun’s article “Why AI systems don’t learn and what to do about it”.
Info.json:
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
"features": {
"observation.images.cam_high": dtype=video,
"observation.images.cam_left_wrist": dtype=video,
"observation.images.cam_low": dtype=video,
"observation.images.cam_right_wrist": dtype=video,
"observation.state": dtype=float32,
"action": dtype=float32,
"episode_index": dtype=int64,
"frame_index": dtype=int64,
"timestamp": dtype=float32,
"next.done": dtype=bool,
"index": dtype=int64,
"task_index": dtype=int64
}
The data_path and video_path fields define where episode data and video recordings are stored within the dataset directory structure.
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4"
The videos folder is used for storing video recordings of episodes in MP4 format. Each episode is typically synchronized with one or more camera views (e.g., wrist, head, or external cameras).
The data folder contains all other numerical and sensor information, such as robot joint states, actions, timestamps, and episode metadata. This information is stored in Parquet format and structured as a time series.
Example row structure:
observation.state action episode_index frame_index timestamp next.done index task_index
[-0.0015339808, -0.96334... [-0.013805827, -0.9526021... 0 0 0.00 False 0 0
Here is s brief explanation of each filed in the parquet
This keeps the trajectory of the robot state, i.e. the time series of all joint angles at each time point, so in the observation.state column, we can see these numbers:
[-0.00153398, -0.96334 , 1.1734953 , 0.00153398, -0.2960583 ,
-0.00153398, -0.04903939, -0.00460194, -0.96334 , 1.1734953 ,
-0.00153398, -0.2960583 , -0.00153398, -0.0205711 ]The meaning of those numbers is explained in the names section of info.json. So basically, in our case of the two-arm ALOHA robot, we only have servo motor angles, which completely define the robot state:
"names": {
"motors": [
"left_waist",
"left_shoulder",
"left_elbow",
"left_forearm_roll",
"left_wrist_angle",
"left_wrist_rotate",
"left_gripper",
"right_waist",
"right_shoulder",
"right_elbow",
"right_forearm_roll",
"right_wrist_angle",
"right_wrist_rotate",
"right_gripper"
]
}These are joint angles (positions) in radians of each servo motor in the current robotic setup. In our example, since this is an ALOHA bimanual robot dataset, we have two arms, left and right, with corresponding motors.
Actions here are commanded joint positions — meaning the controller instructs the motors to move to these target angles.
[-0.01380583, -0.9526021 , 1.1688933 , -0.00153398, -0.30679616,
-0.00153398, -0.12270375, -0.0076699 , -0.960272 , 1.1627575 ,
-0.00306796, -0.30526218, -0.00153398, -0.15765645]So, in other words, given the current joint angles in observation.state, the robot is commanded to transition to the next joint position specified in the action row.
observation.state_t (current joint angles)
action_t (desired joint angles)
# at next time t+1
observation.state_t_next (next joint angles)Note that VLA is different from imitation learning in that it also includes a language command. This is where task_index comes into play. We can see that in our data it is always zero. It is actually an index of the language command, and we can find the corresponding command in tasks.jsonl:
{"task_index": 0, "task": "Pick up the plastic cup and open its lid."}Now we can see that LeRobot also supports VLA training.
episode_000X.parquet file.t + 1 is terminal (see the next paragraph).The differences between v2.0 and v3.0 are mostly cosmetic, and their purpose is primarily optimization for large-scale datasets.
Briefly
Purpose of such chunking is
Also, in modern training setups, you do not train episode-by-episode anymore. Instead, you sample random frames or short sequences. Because of this, chunking across multiple episodes is a more suitable data organization.
So, overall the LeRobot v.30 folder structure is as follows:
.
├── data
│ └── chunk-000
│ └── file-000.parquet
├── meta
│ ├── episodes
│ │ └── chunk-000
│ │ └── file-000.parquet
│ ├── info.json
│ ├── stats.json
│ └── tasks.parquet
├── README.md
└── videos
├── observation.images.cam_high
│ └── chunk-000
│ └── file-000.mp4
├── observation.images.cam_left_wrist
│ └── chunk-000
│ └── file-000.mp4
├── observation.images.cam_low
│ └── chunk-000
│ └── file-000.mp4
└── observation.images.cam_right_wrist
└── chunk-000
└── file-000.mp4
Note, that there is also an additional episode-level metadata structure (meta/episodes), because episodes are no longer stored as individual Parquet files. Instead, multiple episodes are packed into a single chunked Parquet file, so explicit indexing is required to map each row back to its corresponding episode.
The structure described above is ideal for imitation learning because, in imitation learning, the agent reacts at every time step.
Each time step provides a training example, and we can train the policy as follows:
function to learn: observation_t → action_t
loss = || predicted_action - action_t ||In imitation learning, we do not need a reward signal and do not have to wait until the end of the episode. However, we do need teleoperation or expert demonstration data. This type of dataset is well-suited for simple policies like ACT or VLA. VLA models are also typically trained in an imitation learning setup, with the addition of a language command associated with the entire episode.
For the RL situation is a bit different: we need the data of the following structure
(state_t, action_t, reward_t, state_{t+1})The key addition here is the reward signal. In RL, we do not have access to expert demonstrations that define the correct action at each step. So instead of asking “what is the correct action?”, we ask: “How good was this action based on what happened next?”
(state_t, action_t) → (reward_t, state_{t+1})So our reward is essentially a proxy for “how good was that step for the overall task?”
One complication is that the reward is not something you “get for free”; you have to design or measure it from sensors, environment state, or task success signals or we can artificially calculate reward by one of the following technics.
For example it might be distance-based reward - by computing distance between end-effector and target Reward or something like that. Or Reward also might be computed from images or even with only final reward, RL can propagate it backward.
Let’s explore it using an example of the cube_to_bowl_5 dataset for a single robotic arm.
It is very similar to LeRobot v2.0, with several distinctions. We can think of it as GR00T LeRobot, as described here.
The main difference compared to LeRobot 2.0 is the introduction of a new metadata file:
modality.json # -> GR00T LeRobot specificThis file provides detailed metadata about state and action modalities.
GROOT dataset looks like the following:
.
├── data
│ └── chunk-000
│ ├── episode_000000.parquet
│ ├── ...
│ └── episode_000004.parquet
├── meta
│ ├── episodes.jsonl
│ ├── info.json
│ ├── modality.json
│ ├── relative_stats.json
│ ├── stats.json
│ └── tasks.jsonl
└── videos
└── chunk-000
├── observation.images.front
│ ├── episode_000000.mp4
│ ├── ...
│ └── episode_000004.mp4
└── observation.images.wrist
├── episode_000000.mp4
├── ...
└── episode_000004.mp4
The most important addition are:
episodes_stats.jsonl (used in LeRobot 2.0), GR00T introduces:
The change in statistics files is mostly cosmetic. In general, statistics provide aggregated values (min, max, mean, etc.) for each joint dimension or feature across time.
The modality.json file defines how raw data should be interpreted by the training pipeline.
It tells the training pipeline what signals exist, how they are structured, and how to interpret raw data fields.
In plain words it tells the training pipeline the structure of your robotics system: how many arms, with how many degrees of freedom, how many cameras etc.
{
"state": {
"single_arm": {
"start": 0,
"end": 5
},
"gripper": {
"start": 5,
"end": 6
}
},
"action": {
"single_arm": {
"start": 0,
"end": 5
},
"gripper": {
"start": 5,
"end": 6
}
},
"video": {
"front": {
"original_key": "observation.images.front"
},
"wrist": {
"original_key": "observation.images.wrist"
}
},
"annotation": {
"human.task_description": {
"original_key": "task_index"
}
}
}LeRobot v2.0 provides a clean and interpretable representation of robotic trajectories, while v3.0 improves scalability through chunked storage and streaming-oriented design.
GR00T extends this idea further by explicitly introducing modality-level definitions that make datasets more portable across different robot embodiments and sensing setups.