Let colors get used on the crucial parts of the image.

Edit your starting image background and make it a sepia tone, black and white, blurĀ the colors out or delete it. If thereĀ are lots of colors, your images may be going to maximum colors before it makes the part of the photo you care about.

A colorful object in the background that keeps using up all the colors