By use of the measure, the backflow of information presented recently, we study the non-Markovianity of the dynamics for a two-level system interacting with a zero-temperature structured environment via amplitude-phase coupling. In the limit of weak coupling between the system and its reservoir, the time-local non-Markovian master equation for the reduced state of the system is derived. Under the secular approximation, the exact analytic solution is obtained. Numerical simulations show that the amplitude and phase dampings can produce destructive interference to the backflow of information, leading to the weaker non-Markovianity of the compound dynamics compared with the dynamics of a single amplitude or phase damping model. We also study the characteristics of the initial-state pairs that maximize the backflow of information.