Let colors get used on the crucial parts of the image.

Edit your starting image background and make it a sepia tone, black and white, blur the colors out or delete it. If there are lots of colors, your images may be going to maximum colors before it makes the part of the photo you care about.

A colorful object in the background that keeps using up all the colors