It depends a little bit on how you define “human impact.” There has been a lot written about the ethical problems of habitat restoration. You can read a summary of the ethical problems some have seen in the article on “Environmental Restoration” in the Encyclopedia of Geography. One of the main sources of that argument is Robert Elliot’s Faking Nature, which we have in print and as an ebook .
If you search the databases for terms like ((“environmental restoration” OR “ecological restoration” OR “habitat restoration”) AND controvers*) you'll find a lot of ebooks and articles that will cover some of the problems people see in restoration projects (even if the treatment of the topic in the book or article as a whole is not negative). You could also add search terms such as “unintended consequences”. As an example, one article mentions in its abstract that "restoration efforts in remote ecosystems might do more harm than good, and the effort required for effective restoration might be greater than easily justified given the shortfall of resources for restoring more heavily impacted ecosystems." Another article shows how tricky and error-prone the whole enterprise can be. I also saw an article noting that restoration can have side effects: restoration of wetlands can result in restoring habitat to disease-bearing mosquitos. There's an adverse human effect.
So, even if it's tough to find much literature arguing against habitat restoration per se, you can extract the arguments against it from the mentions of controversy, conflict, and dangers in these articles.
Hope that helps!