摘要： In this paper, the capacity of a free-space optical (FSO) communication over radio frequency (RF) could potentially be increased by the simultaneous transmission of multiple orbital angular momentum (OAM) based on spatial mode multiplexing (SMM) as an additional effective degree of freedom (EDOF) and quadrature amplitude modulation (QAM). This paper describes using hybrid RF/FSO-OAM based on multiple-input multiple-output (MIMO)/SMM using M-ary modified pulse position modulation (MPPM) and spatial PPM (SPPM) for potentially enhancing capacity in wireless communication systems. We present an analytical system design concerning OAM multiplexing and MIMO/SMM processing in free space communications channel under atmospheric turbulence (AT). Here, we assume a new architecture that closes the gap in speeds between millimeter-wave (mm-wave) wireless and optical links. In this study, we highlight recent advances in the use of OAM multiplexing for high-capacity FSO and mm-wave communications. We have developed the MPPM in FSO communication over RF using the source Gaussian model and SMM by proposing a new version of hybrid SPPM and MPPM. The novelty approach presents performance enhancement of the acquisition, pointing, tracking position and AT mitigation of link. We propose to use SMM combined with OAM-QAM based MIMO communications system to mitigate both weak and strong turbulence distortions. Simulation results show that the capacity of OAM-based MIMO system outperforms the capacity of the conventional MIMO system when the propagation distance is larger than a specific threshold. The use of transmitter lenses could enhance the OAM beams with a larger mode spacing (MS) of 2 (OAM l +1, l +3, l +5, l +7 transmitted) shows a lower power penalty (PP) when the inter-channel crosstalk overwhelms a larger lateral displacement of about 2 mm. Using the lenses at the transmitter to focus OAM beams could reduce power loss and PP in OAM-based FSO links and that this improvement might be more significant for higher-order OAM beams and provide high power-efficiency. With a larger of the transmitter and receiver 8 cm, 10 cm aperture size respectively, the system with mode spacing of 2 (OAM l +1, l +3, l +5, l +7 transmitted) and mode spacing of 3 (OAM l +1, l +4, l +7, l +10 transmitted) of OAM beams based on MIMO should fulfill lower-power penalty of 2.2 dB, 3.4 dB, as respectively and could enhance system robustness under angular error but degrade tolerance of lateral displacement. The proposed system provides an excellent signal-to-noise ratio (SNR) for the capacity in the regime of strong atmospheric turbulence. In this way, the maximization of FSO communication and RF wide, the data capacity enhancement, and it is considered a massive solution in the bandwidth provision for future access networks. This work could be beneficial to the practical implementation of OAM-MIMO/SMM multiplexed RF/FSO links.