While the average taxpayer faced a less than 1 percent chance of an audit in the 2014 fiscal year, those making $200,000 had twice that rate, and the higher the income, the more likely the auditors were to take a look.

Those with annual incomes of at least $10 million had a one in four chance of facing an audit in the 2013 fiscal year, which dropped to a one in six chance by last year, the agency's inspector general said in a report late last week.

It's all very much by design. Wealthy taxpayers are more likely to use complex financial arrangements to manage their money, and audits are needed to keep them in compliance and paying their fair share under the law.

But the inspector general said the agency needs to home in on whom it considers to be high-income taxpayers deserving of special scrutiny, to make sure it's really going after the people who are hiding the most money and where the agency gets the best return on investment.