Research on Cooperative Output Regulation of Heterogeneous Directed Multi-Agent Systems: A Fully Distributed Model-Free Reinforcement Learning Framework

Background

In recent years, the study of distributed control and optimization has demonstrated broad application prospects in smart transportation, smart grids, distributed energy systems, and other fields. These systems often require collaboration among multiple agents to complete specific tasks. One fundamental research topic in this field is the cooperative output regulation (COR) problem. COR aims to design appropriate control protocols to ensure that all agents in a multi-agent system track a reference signal and achieve zero tracking errors. However, existing methods typically require precise knowledge of the dynamic models of the agents, which is often difficult to acquire in real-world scenarios due to a complex environment, high coupling of non-linearities, or prohibitive measurement costs.

Additionally, communication networks in multi-agent systems are often directional (information transfer is unidirectional), which further complicates the study of this problem. Existing methods often focus on undirected graph structures, whereas directed multi-agent systems are usually more complex, particularly under conditions of unknown models or limited information access. Thus, achieving fully distributed, event-triggered mechanisms (ETM), and model-free control designs in the context of directed heterogeneous multi-agent systems has become an urgent challenge to address.

Paper Source

The paper, titled “Cooperative Output Regulation of Heterogeneous Directed Multi-Agent Systems: A Fully Distributed Model-Free Reinforcement Learning Framework,” is authored by Xiongtao Shi, Yanjie Li (corresponding author), Chenglong Du (corresponding author), Huiping Li, Chaoyang Chen, and Weihua Gui. The authors are affiliated with institutions such as Harbin Institute of Technology (Shenzhen), Central South University, Northwestern Polytechnical University, and Hunan University of Science and Technology. The paper was published in Science China Information Sciences, February 2025, Volume 68, Issue 2, Article ID 122202. It proposes a fully distributed control framework based on model-free reinforcement learning to address the COR problem in directed heterogeneous multi-agent systems where dynamic models are unknown, and only local communication is available.

Research Workflow

1. Research Content Summary

The paper studies the COR problem under two scenarios: 1. Scenario 1: The exosystem is globally accessible to all agents. - In this scenario, the authors design an Augmented Algebraic Riccati Equation (AARE) and solve the feedback gain matrix using a model-free reinforcement learning algorithm.

1. Scenario 2: The exosystem is only accessible to its adjacent followers.
   • In this scenario, the researchers further design distributed observers for each agent and propose an observer-based adaptive event-triggered control protocol.

Through these two scenarios, the research aims to: - Eliminate dependence on model dynamics; - Reduce communication load and computational cost using event-triggered control; - Address the COR problem in a fully distributed manner within directed graph structures.

2. Research Process and Algorithm Details

Scenario 1: Exosystem Accessible to All Agents Globally

In this scenario, the researchers construct internal models for each agent and design a control protocol:

1. Construction of the Internal Model:

   • The external system’s state information is integrated into the internal model’s state update, introducing an inherent feedback gain matrix.

2. Reinforcement Learning to Solve Feedback Gain Matrix:

   • The authors define the Augmented Algebraic Riccati Equation (AARE), whose solution directly provides feedback gain for the control protocol.
   • An iterative model-free reinforcement learning algorithm is introduced, utilizing input-output data to compute the feedback gain matrix online.

The algorithm’s core involves matrix iteration formulas and Lyapunov stability analysis. The proposed reinforcement learning algorithm guarantees the convergence of the feedback gain matrix to the target value through structured exploration noise and update criteria.

Scenario 2: Exosystem Locally Accessible

To handle more complex scenarios where only local access to the exosystem is possible, the researchers introduce a fully distributed event-triggered control framework:

1. Introduction of Distributed Observers:

   • Using the exosystem state and local neighbor information, distributed observers are constructed to estimate the exosystem state.

2. Event-Triggered Mechanism:

   • A novel adaptive event-triggered function is designed that activates inter-agent information sharing on-demand, significantly reducing communication frequency.
   • Triggering is based on reaching preset thresholds for estimation errors.

3. Fully Distributed Control Protocol:

   • Without relying on the global Laplacian matrix, the researchers complete the design of a control protocol using adaptive gains and a novel graph-based Lyapunov function.

4. Theoretical Proof and Robustness Verification:

   • Rigorous analysis is provided on the convergence of adaptive parameters and event-triggering frequency.
   • A coupled system Lyapunov function is constructed to prove the global asymptotic stability of the control protocol.

3. Simulation Design

The simulation involved a directed graph network of four agents, where the dynamics of all agents were unknown. The researchers conducted two sets of experiments under the two scenarios to validate the proposed method:

1. Experiments Based on Scenario 1:

   • Algorithm 1 successfully learned feedback gain matrices close to their model-based solutions.
   • The agents achieved rapid tracking of the reference signals generated by the exosystem.

2. Experiments Based on Scenario 2:

   • The effectiveness of the adaptive event-triggered mechanism was demonstrated. Compared to the traditional static event-triggered mechanism, the adaptive mechanism significantly reduced communication frequency while achieving similar control performance.
   • The average event-trigger intervals under the adaptive mechanism increased by 1.5 to 3 times compared to the static mechanism.

Research Findings and Significance

1. Major Findings

This paper achieves significant breakthroughs in the COR problem of directed heterogeneous multi-agent systems. By constructing internal models and designing model-free reinforcement learning algorithms, it realizes fully distributed control protocol learning online. Combined with an adaptive event-triggered mechanism, the proposed approach effectively avoids reliance on global information and frequent communication.

2. Academic Value

The proposed method not only enriches theoretical research in distributed control of multi-agent systems but also provides a universal solution for similar collaboration problems under complex scenarios. On an application level, the method simplifies dependency on model information during engineering implementation, making it suitable for practical problems such as robot formation control, UAV group collaboration, and distributed energy system regulation.

3. Highlights

• Elimination of Model Dependency: The reinforcement learning algorithm constructs the feedback matrix using input-output data without requiring precise dynamic models.
• Study of Directed Multi-Agent Systems: The research extends distributed control scenarios to more complex directed graph structures.
• Innovative Event-Triggered Mechanism: The adaptive ETM overcomes resource wastage caused by frequent communication.

Outlook

Future research directions include investigating communication topologies with dynamic changes and applying the proposed method to practical multi-agent systems, such as robot swarms and vehicle formations, to verify its real-world applicability.