Abstract: In order to analyze the influence of the nonlinearity of the radio frequency receiver in the multi-antenna system on the performance of the multi-antenna communication system, we study the single-input single-output (SISO) and multiple-input multiple-output (MIMO) systems. After the research, it is pointed out that the original definition of the compromise between nonlinear diversity and multiplexing is no longer applicable when considering the nonlinearity of radio frequency receivers, and the applicable definition and its physical meanings are given. Through composite channel modeling, the definition of diversity gain and the expression of outage probability under non-ideal conditions are theoretically deduced, and the relationship curve between diversity gain and multiplexing gain of multi-antenna system considering the nonlinearity of radio frequency receiver is obtained by simulation, and the relationship between diversity gain and multiplexing gain is obtained. The growth law of the target rate in the compromise with reuse. Theory and simulation show that the traditional definitions of diversity gain and multiplexing gain are not suitable for the actual situation of considering the nonlinearity of the receiver, and the compromise effect of the nonlinearity of the RF receiver in the SISO system on the diversity and multiplexing is the same as the ideal situation; in the MIMO system, the original diversity gain and multiplexing gain need to be modified to obtain the optimal compromise between diversity and multiplexing.