Ismael Espinoza Jaramillo
Portrait of Ismael Espinoza Jaramillo

Ismael Espinoza Jaramillo

Robot learning researcher · Ph.D. candidate, Kyung Hee University (degree February 2027)

Available from March 2027 · Gyeonggi-do, South Korea · open to relocation

I train policies for a 44-DoF robot with two arms and two dexterous hands in simulation and run them on the real robot, with no real-robot training data.

  • 4 long-horizon tasks on the real robot: 18 to 30 of 35 trials each (Bimanual Sweeping 30/35)
  • 72 to 84% real-robot success from two RGB cameras, no pose estimator
  • IEEE RA-L 2024 co-first author · Korean patent 10-2943203 granted

The whole path from data to hardware: glove and VR teleoperation, digital twins in Isaac Lab, transformer policies trained with imitation and reinforcement learning, and ROS 2 deployment with 6D object tracking or RGB-only vision. Before robot learning: industrial automation in Ecuador (PLCs, LiDAR navigation).

Bimanual Sweeping Real robot Direct transfer · 30/35 successes · 2× speed

On the real robot

Each policy was trained only in the digital twin and run on the robot without real-robot training data.

Cup Handover Real robot Direct transfer · 29/35 successes · 3× speed
Tube Handover, RGB only Real robot Distilled student · 84.0% success · 1.5× speed
Bimanual Pick and Place, RGB only Real robot Distilled student · 72.0% success · real time

News

Research

Dissertation: A Real2Sim2Real System for Bimanual Long-Horizon Dexterous Manipulation via Temporal-Context Transformer Reinforcement Learning (Kyung Hee University, submitted October 2026).

Diagram of the Real2Sim2Real loop: physical robot and operator in the real world; digital twin, policy with demonstrations, training in parallel twins, and two transfer paths back to the robot
The Real2Sim2Real loop, closed on two robot configurations: build the digital twin from the real robot, seed the policy with teleoperated demonstrations, train it in parallel twins, then return to the robot by direct or distilled transfer. Click to enlarge.
Table-mounted and upper-humanoid robot configurations shown beside their digital twins
Table-mounted (top) and upper-humanoid (bottom) configurations beside their digital twins.

1 · Real2Sim

A digital twin built from the real robot

The robot has a UR3e and a UR3 arm, each with a 16-DoF Allegro Hand V4: 44 joints controlled by one policy. Its digital twin uses the robot description, object meshes scanned with a phone and cleaned into collision geometry, and drive gains calibrated against motion recorded on the hardware. The same system was built for two configurations: table-mounted and upper-humanoid.

Object reconstruction figure

Real robot The glove interface driving the real robot · 2× speed. The training demonstrations are recorded in the digital twin.

2 · Demonstrations

Thirty teleoperated demonstrations become three hundred

Demonstrations are recorded inside the digital twin. The operator wears two Manus gloves and three VIVE trackers; a learned retargeting network (GeoRT) maps the fingertips to the Allegro joints, and the wrist poses drive the arms through inverse kinematics. The thirty demonstrations per task are split into subtask segments, recombined into new episodes and replayed in the twin. Only physically valid episodes are kept, giving 300 per task.

Teleoperation pipeline · Finger retargeting · Demonstration stitching

3 · Policy

TC-TRL: two arms, one memory

TC-TRL in 30 seconds: remember, decide for the right arm, then the left arm knowing the right.

Each arm with its hand is an agent. A temporal-context encoder attends over a window of recent observations and actions, so the policy knows which step of a long task is under way; an arm-conditioned decoder chooses the left arm's action knowing the right arm's. The policy is pretrained by behavior cloning on the demonstrations, then fine-tuned with PPO in parallel copies of the twin with domain randomization (upper-humanoid configuration: 300 demonstrations, 500 environments, about ten hours per task on one RTX 5070 Ti).

Simulation On the five long-horizon benchmark tasks of the table-mounted configuration (Isaac Gym twin), 89.9% mean success, against 59.0% for the best transformer baseline and 26.6% for the best PPO baseline.

IEEE RA-L 2024 (co-first author) · Korean Patent 10-2943203 · Architecture

4a · Direct transfer

The same policy, unchanged, on the real robot

A ROS 2 bridge runs the trained policy at 10 Hz. An RGB-D camera in front of the robot and a 6D pose pipeline (detection, segmentation, FoundationPose) supply the object poses the policy saw in the twin, and the 44 joint targets are smoothed and clamped before they reach the drivers.

Digital twin Cup Handover · 3× speed
Real robot Cup Handover · 3× speed
Direct transfer: success on the real robot (35 trials per task)
TaskConfigurationDigital twin, TC-TRLReal robot, TC-TRLReal robot, imitation only
Tube Handover (H1)table-mounted82.9%27/3577.1%4/3511.4%
Cup Handover, flip and place (H2)table-mounted57.1%18/3551.4%2/355.7%
Cup Handover (H3)upper-humanoid93.8%29/3582.9%13/3537.1%
Bimanual Sweeping (H4)upper-humanoid90.9%30/3585.7%7/3520.0%

TC-TRL loses 5 to 11 points from the twin to the real robot; the same architecture trained by imitation alone reaches at most 37.1%. Source: dissertation, Chapter 4. Pose estimation figure · The two transfer paths

Real robot Tube Handover (H1), table-mounted · 1.5× speed
Real robot Cup Handover, flip and place (H2), table-mounted · 1.5× speed

4b · Distilled transfer (TMAD)

Dropping the pose estimator: a student that acts from two RGB cameras

The trained policy stays in the twin as a frozen teacher. A student that sees only two RGB images and the robot's proprioception is distilled from it online, in the states the student itself reaches, with randomized rendering. On the robot the student runs at 30 Hz with no pose estimator, no object model and no extrinsic camera calibration.

TMAD overview: a frozen teacher in the digital twin, online distillation of a student from rendered and randomized images, and deployment of the student on the robot
TMAD: teacher in the twin, online distillation, deployment of the student only.
Real robot Pouring Water, RGB only · 2× speed
Distilled transfer: success on the real robot, upper-humanoid (mean of three evaluation runs per task)
TaskRGB student (TMAD)Single-agent student
Bimanual Pick and Place72.0%28.0%
Tube Handover84.0%24.0%
Pouring Water78.1%34.2%
Bimanual Sweeping81.2%75.9%

Source: dissertation, Chapter 5. The single-agent student (DextrAH-RGB style) has the same encoder and training but no temporal multi-agent core. The TMAD paper under review (second author) reports the first two tasks, each over three runs of 25 trials.

Code

Public repositories on GitHub. Results are as reported in each repository's README.

dexterous-pick-and-place

FoundationPose perception and force-closure grasp synthesis (BODex on cuRobo) driving an Allegro Hand on a UR3, in Isaac Lab and on the real robot.

Real robot 94 of 100 executed trials succeeded across 10 objects. Simulation 180 of 200 episodes, both arms.

Python · Isaac Lab · cuRobo · FoundationPose

dual-arm-allegro-ros2

ROS 2 Jazzy drivers for the bimanual torso: UR3 and UR3e arms, two Allegro Hand V4 over CAN, three RealSense D415 cameras, URDF of the stand and table, and bridges that take plain joint-position topics.

44 joints in one /joint_states at 100 Hz; the configuration the lab's robot runs daily.

C++ · ROS 2 · ros2_control · Docker

allegro-hand-retargeting

Real-time Manus glove to Allegro Hand retargeting for both hands, with a learned geometric model (GeoRT, trained without paired data) and a rule-based baseline, streamed to the hands through a 60 Hz ROS 2 bridge.

About 0.7 ms per hand per forward pass on an RTX 5070 Ti.

C++ · Python · PyTorch · ROS 2

foundationpose-ros2-tracker

Zero-shot 6D object tracking in ROS 2 from a mesh and an object name: YOLO-World detects, SAM 2 segments, FoundationPose registers and tracks, and world-frame poses are published for manipulation.

Includes a port of FoundationPose's CUDA extensions to Blackwell GPUs (CUDA 12.8). MIT license.

Python · ROS 2 · CUDA · SAM 2

quest3-dexterous-teleop

Meta Quest 3 hand-tracking teleoperation of the dual-arm, two-hand robot in Isaac Lab, with demonstration recording and a converter to a fixed-length imitation-learning dataset.

Simulation Teleoperation loop at about 30 Hz; 44-D actions.

Python · Isaac Lab · CloudXR · WebXR

FoldingACT

Vision and touch imitation policy for bimanual paper folding, built for the IROS 2026 Robotic Origami Challenge: an ACT-style action-chunking transformer over four cameras (DINOv3) and ten fingertip tactile pads (Sparsh), predicting 65-D joint targets.

Built the data pipeline for the 398 GB LeRobot dataset, an Isaac Sim mirror, and the inference service the organizers ran.

Python · PyTorch · LeRobot · Isaac Sim

Publications

4 journal articles, 7 international conference papers, 1 granted patent and 2 pending. 153 citations on Google Scholar (October 2026).

Selected

Robot handing a tube between its hands

TMAD: Temporal Multi-Agent Distillation for Bimanual Long-Horizon Manipulation

S. Park, I. Espinoza, G. Kim, J. Y. Kim, K. Lee, T.‑S. Kim

Under review, 2026

Distills the state-based bimanual teacher into a stereo-RGB student that needs no 6D pose estimator; on the real robot, 72.0% on Pick-and-Place and 84.0% on Handover.

Results

Upper-humanoid robot sweeping with a brush and a dustpan

A Semi-Humanoid Real2Sim2Real System for Bimanual Long-Horizon Manipulation

Y.‑H. Choi, I. Espinoza, K. Lee, S. Park, J. Y. Kim, J.‑H. Oh, W. H. Lee, T.‑S. Kim

International Conference on Ubiquitous Robots (UR), Osaka, 2026 · in press

The Real2Sim2Real system on the upper-humanoid configuration: digital twin with real-time object tracking, stitched demonstrations, TC-TRL, and zero-shot transfer of Cup Handover and Bimanual Sweeping.

Results

Table-mounted robot handing a tube to a person

A Real2Sim2Real Transformer Policy System for Bimanual R2R2H Tasks

I. Espinoza, Y.‑H. Choi, K. Lee, J.‑H. Oh, K. J. Pahk, T.‑S. Kim

ACM/IEEE HRI 2026 Companion, Edinburgh, pp. 716–720

Glove-and-tracker teleoperation, a calibrated twin and a transformer policy for robot-to-robot-to-human handover.

Paper · Results

Five long-horizon bimanual tasks in simulation

Bimanual Long-Horizon Manipulation via Temporal-Context Transformer RL

J.‑H. Oh*, I. Espinoza Jaramillo*, D. Jung, T.‑S. Kim  *equal contribution

IEEE Robotics and Automation Letters, 9(12), 2024 · presented at IROS 2025

TC-TRL: a temporal-context multi-agent transformer pretrained by behavior cloning and fine-tuned with PPO for two-arm, two-hand long-horizon tasks.

Paper

Planner architecture combining language, vision and memory

Multi-modal Affordance Planner with Temporal-Context Action Policy for Long-Horizon Bimanual Robot Manipulation

J.‑H. Oh, D. Jung, I. Espinoza, Y.‑H. Choi, Y. Kim, D. Shin, J. Moon, W. Kim, T.‑S. Kim

IEEE ICRA 2026, Vienna, pp. 637–644

A retrieval-augmented LLM planner generates plans, demonstrations and rewards for a temporal-context action policy.

Paper

System overview: wearable exoskeleton sensors, data preprocessing, deep learning models and hardware platforms

Real-time Human Activity Recognition with IMU and Encoder Sensors in Wearable Exoskeleton Robot via Deep Learning Networks

I. Espinoza Jaramillo*, J. G. Jeong*, P. Rivera Lopez, C.‑H. Lee, D.‑Y. Kang, T.‑J. Ha, J.‑H. Oh, H. Jung, J. H. Lee, W. H. Lee, T.‑S. Kim  *equal contribution

Sensors, 22(24), 9690, 2022

My first line of research: deep networks for real-time activity recognition from exoskeleton sensors, running on an embedded board.

Paper

Patents

Assignee: Kyung Hee University Industry-Academic Cooperation Foundation.

Full publication list

Journal articles

  • J.‑H. Oh*, I. Espinoza Jaramillo*, D. Jung, T.‑S. Kim. Bimanual long-horizon manipulation via temporal-context transformer RL. IEEE Robotics and Automation Letters, 9(12), 10898–10905, 2024. doi *equal contribution
  • I. Espinoza Jaramillo, C. Chola, J.‑G. Jeong, J.‑H. Oh, H. Jung, J.‑H. Lee, W. H. Lee, T.‑S. Kim. Human activity prediction based on forecasted IMU activity signals by sequence-to-sequence deep neural networks. Sensors, 23(14), 6491, 2023. doi
  • P. Rivera Lopez, J.‑H. Oh, J. G. Jeong, H. Jung, J. H. Lee, I. Espinoza Jaramillo, C. Chola, W. H. Lee, T.‑S. Kim. Dexterous object manipulation with an anthropomorphic robot hand via natural hand pose transformer and deep reinforcement learning. Applied Sciences, 13(1), 379, 2023. doi
  • I. Espinoza Jaramillo*, J. G. Jeong*, P. Rivera Lopez, C.‑H. Lee, D.‑Y. Kang, T.‑J. Ha, J.‑H. Oh, H. Jung, J. H. Lee, W. H. Lee, T.‑S. Kim. Real-time human activity recognition with IMU and encoder sensors in wearable exoskeleton robot via deep learning networks. Sensors, 22(24), 9690, 2022. doi *equal contribution

International conference papers

  • J.‑H. Oh, D. Jung, I. Espinoza, Y.‑H. Choi, Y. Kim, D. Shin, J. Moon, W. Kim, T.‑S. Kim. Multi-modal affordance planner with temporal-context action policy for long-horizon bimanual robot manipulation. IEEE ICRA, Vienna, 637–644, 2026. doi
  • I. Espinoza, Y.‑H. Choi, K. Lee, J.‑H. Oh, K. J. Pahk, T.‑S. Kim. A Real2Sim2Real transformer policy system for bimanual R2R2H tasks. Companion Proceedings of ACM/IEEE HRI, Edinburgh, 716–720, 2026. doi
  • Y.‑H. Choi, I. Espinoza, K. Lee, S. Park, J. Y. Kim, J.‑H. Oh, W. H. Lee, T.‑S. Kim. A semi-humanoid Real2Sim2Real system for bimanual long-horizon manipulation. International Conference on Ubiquitous Robots (UR), Osaka, 2026, in press.
  • J.‑H. Oh*, I. Espinoza Jaramillo*, D. Jung, Y.‑H. Choi, K. J. Pahk, W. H. Lee, W. Kim, T.‑S. Kim. Bimanual long-horizon lifecare robotics with temporal context LLM planner and transformer RL. IEEE EMBC, Copenhagen, 2025. doi *equal contribution
  • J. H. Lee, J.‑H. Oh, I. Espinoza, D. Jung, Y.‑H. Choi, W. H. Lee, W. Kim, T.‑S. Kim. Counterfactual multi-agent reinforcement learning for long-horizon medical assistive tasks with dual-arm robot. IEEE EMBC, Copenhagen, 2025. doi
  • H. Jung, J.‑H. Oh, I. Espinoza Jaramillo, D. Jung, W. H. Lee, T.‑S. Kim. Multi-agent AMIX-DAPG of dual-arm robot for long horizon lifecare tasks. ICAIIC, Fukuoka, 866–870, 2025. doi
  • J. G. Jeong, J.‑H. Oh, H. Jung, J. H. Lee, I. Espinoza Jaramillo, C. Chola, W. H. Lee, T.‑S. Kim. Long-horizon manipulation by a single-arm robot via sub-goal network based hierarchical reinforcement learning. ICBET, Tokyo, 88–92, 2023. doi

Symposium poster

  • J.‑H. Oh, I. Espinoza, H. Jung, J. H. Lee, S. Kim, D. Jung, W. H. Lee, W. Kim, T.‑S. Kim. DA-TRL: Demonstration-augmented transformer reinforcement learning for long-horizon bimanual tasks by dual-arm robot. IEEE/SICE SII, Ha Long, 2024.

Under review

  • S. Park, I. Espinoza, G. Kim, J. Y. Kim, K. Lee, T.‑S. Kim. TMAD: Temporal multi-agent distillation for bimanual long-horizon manipulation. 2026.

Domestic conference papers (KICS, Korea)

  • D. Jung, J.‑H. Oh, I. Espinoza Jaramillo, Y.‑H. Choi, K. Lee, T.‑S. Kim. Daily human activity recognition using joint angle feature based LSTM. KICS Fall Conference, 2024.
  • Y.‑H. Choi, I. Espinoza Jaramillo, J.‑H. Oh, D. Jung, K. Lee, T.‑S. Kim. Robot-human handover of objects via human hand pose recognition. KICS Fall Conference, 2024.
  • J. H. Lee, H. Jung, J.‑H. Oh, I. Espinoza Jaramillo, S. Kim, D. Jung, T.‑S. Kim. Development of dual-arm robot intelligence for long-horizon medical assistive tasks via collaborative reinforcement learning. KICS Fall Conference, 2023.
  • H. Jung, J. H. Lee, J.‑H. Oh, I. Espinoza Jaramillo, S. Kim, D. Jung, T.‑S. Kim. Dual-arm robot manipulation of long-horizon lifecare tasks via multi-agent VAMIX-DAPG reinforcement learning. KICS Fall Conference, 2023.
  • J.‑H. Oh, J. G. Jeong, I. Espinoza Jaramillo, H. Jung, J. H. Lee, C. Chola, T.‑S. Kim. Autonomous object grasping with an anthropomorphic robot hand via demonstration augmented deep reinforcement learning. KICS Fall Conference, 2022.
  • J. G. Jeong, H. Jung, J. H. Lee, J.‑H. Oh, I. Espinoza Jaramillo, C. Chola, T.‑S. Kim. Solving kitchen long horizon tasks with an anthropomorphic robot hand via sub-goal network-based deep reinforcement learning. KICS Fall Conference, 2022.

Experience & education

Experience

  1. Mar 2022 – present

    Graduate Researcher (Ph.D.), Robot Learning

    Kyung Hee University, Yongin, South Korea · Advisor: Prof. Tae-Seong Kim

    Designed and built the Real2Sim2Real system of my dissertation: teleoperation and retargeting, Isaac Lab digital twins, and ROS 2 deployment on two robot configurations. Co-developed the TC-TRL policy (co-first author) and the TMAD distillation (second author).

  2. May 2021 – Feb 2022

    Research Assistant, Mobile Robotics

    Universidad de las Fuerzas Armadas (ESPE), Quito, Ecuador

    ROS software stack (Python, C++) for a social-navigation and telepresence robot; BLE indoor localization.

  3. Nov 2020 – May 2021

    Project Development Assistant, Industrial Automation

    Robotron S.A., Quito, Ecuador

    LiDAR and QR-marker forklift navigation and pallet tracking for a sub-zero warehouse; C++ and Python bridges between Siemens PLCs and a Microsoft Azure database; on-site installation.

  4. Aug 2020 – Oct 2020

    Electrical Design Intern

    SEIUS S.A., Quito, Ecuador (remote)

Education

  1. Mar 2022 – Feb 2027 (expected)

    Ph.D. (integrated M.S.–Ph.D.), Electronics and Information Convergence Engineering

    Kyung Hee University, South Korea · major: Biomedical Engineering · GPA 4.27/4.30

  2. Sep 2015 – Feb 2021

    B.Sc., Electronics, Automation and Control Engineering

    Universidad de las Fuerzas Armadas (ESPE), Ecuador · GPA 17.15/20

Honors

  • President's Scholarship, Kyung Hee University, all six semesters (2022–2024)
  • BK21 FOUR Scholarship (2023–2024) and Fellowship (2025)
  • Journal Publication Scholarship, Kyung Hee University (2023)

Skills

Robot learning
RL (PPO, multi-agent), imitation learning, ACT, SmolVLA (in progress), teacher-student distillation, sim-to-real, domain randomization
Perception
FoundationPose, YOLO-World, SAM 2, RGB visuomotor policies, RealSense
Simulation
Isaac Lab, Isaac Sim, Isaac Gym
Software
Python, C++, PyTorch, CUDA, ROS 2, Docker, Linux, Git
Hardware
UR3 and UR3e, Allegro Hand V4, Manus gloves, VIVE trackers, Siemens PLCs
Languages
Spanish (native), English (fluent), Korean (basic)

Contact

Based in Gyeonggi-do, South Korea, and open to research scientist and research engineer roles in robot learning, in Korea or abroad.

Personal
inespinozarir10@gmail.com
Lab
inespinoza24@khu.ac.kr
CV
Espinoza_Ismael_CV.pdf
Scholar
Google Scholar profile
LinkedIn
linkedin.com/in/ismael-e-1629bb229