Abstract: For Joint all-domain operations, this paper proposes a scenario for cooperative operations of maritime unmanned cluster defense systems. Based on deep reinforcement learning, a heterogeneous cluster multi-agent deep deterministic policy gradient（MADDPG）algorithm is proposed. On this basis, the model structure of MADDPG algorithm is designed from the aspects of state space, action space and reward function. The framework of centralized training and decentralized execution allows agents learn collaborative behavior quickly. This paper carried out a simulation for this scenario, and verified that the learned combat unit has the ability to cooperate in combat, making the combat process more intelligent.