In order to analyze the stress and deformation of different permeability of residual oil film in the complex pore, which are affected by the viscous-elasticity of the fluid, the hydrodynamic displacement mechanism is explored from the stand-point of hydrodynamics, that is, the residual oil film displaced by alternating injection of different concentrations of the polymer solution, viscous-elastic fluid flow equation is established in the complex pore by choosing continuity equation, motion equation and the upper convected Maxwell constitutive equation. The flow field is computed by using the method of numerical analysis. Not only the stress and deformation of residual oil film on the different permeability of micro pores, but also the analysis of the flooding mechanism of alternating injection of different concentrations of the polymer solution is got. The results show that the larger the viscous-elasticity of polymer solution is, the bigger the normal deviatoric stress acting on the residual oil film is; the distribution of normal deviatoric stress has the abrupt change. The stronger the viscous-elasticity of the polymer solution is, the bigger the horizontal stress difference acting on the residual oil film is and the more obvious the deformation is; the high-concentration polymer solution is suitable for high-permeability micro pores. Low-concentration polymer solution is suitable for medium and low-permeability micro pores. Alternating injection of polymer solution can improve Volumetric Sweep Efficiency and increase the deformation of residual oil film, which is conducive to enhancing oil recovery.