The mixed convection two-phase flow and heat transfer of nanofluids were addressed within a wavy wall enclosure containing a solid rotating cylinder. The annulus area between the cylinder and the enclosure was filled with water-alumina nanofluid. Buongiorno's model was applied to assess the local distribution of nanoparticles in the host fluid. The governing equations for the mass conservation of nanofluid, nanoparticles, and energy conservation in the nanofluid and the rotating cylinder were carried out and converted to a non-dimensional pattern. The finite element technique was utilized for solving the equations numerically. The influence of the undulations, Richardson number, the volume fraction of nanoparticles, rotation direction, and the size of the rotating cylinder were examined on the streamlines, heat transfer rate, and the distribution of nanoparticles. The Brownian motion and thermophoresis forces induced a notable distribution of nanoparticles in the enclosure. The best heat transfer rate was observed for 3% volume fraction of alumina nanoparticles. The optimum number of undulations for the best heat transfer rate depends on the rotation direction of the cylinder. In the case of counterclockwise rotation of the cylinder, a single undulation leads to the best heat transfer rate for nanoparticles volume fraction about 3%. The increase of undulations number traps more nanoparticles near the wavy surface.