Shadow and variable illumination considerably influence the results of image understanding such as image segmentation, object tracking, and object recognition. The intrinsic image decomposition is to separate the reflectance and the illumination image from an observed image. The intrinsic image decomposition is very useful to remove shadows and then improve the performance of image understanding. In this paper, we present a new shadow removal method based on intrinsic image decomposition on a single color image using the Fisher Linear Discriminant (FLD). Under the assumptions-Lambertian surfaces, approximately Planckian lighting, and narrowband camera sensors, there exist an invariant image, which is 1-dimensional greyscale and independent of illuminant color and intensity. The Fisher Linear Discriminant is applied to create the invariant image. And further the shadows can be removed through the difference between invariant image and original color image. The experimental results on real data show good performance of this algorithm.