A report says ministers may not be able to keep a promise on affordable housing because of the economic downturn.