Abstract: The control problem of the model-free robust optimal formation control of heterogeneous quad-rotor unmanned helicopter cluster is solved with reinforcement learning methods. Each quad-rotor unmanned helicopter is considered as nonlinear and underactuated systems subject to such influential factors as unknown dynamical model and external disturbances, etc. A fully distributed observer is firstly designed. The local information is used to generate the reference signals for the unmanned helicopter cluster to achieve the desired flight formation. Then, a robust optimal position controller and a robust optimal attitude controller are designed via reinforcement learning without requirement of the dynamical information of unmanned helicopter. The effectiveness of the proposed formation control algorithm is verified by the theoretical analysis and simulation.