I'd say Square Enix was more interested in Tomb Raider and Hitman, and in that package DX was just another promising concept with its critcal acclaim in the past. Also, I didn't have the impression DXHR was thought to be unsuccessful. At least during the fiscal year report of DXHR's realease it was said to be a reason that Square Enix didn't suffer that much from the disappointment of the Final Fantasy franchise at that time. It wasn't only a number thing, but also outstanding in art direction and how to concept a large game franchise. Last but not least Square Enix used it to connect their Japanese facilities with western game development know how.
DX is cyberpunk stealth action. You can find successful titles in both areas during the last years (e.g. Metal Gear Solid and Dishonored in the stealth action area). I also don't think the dev team failed on the premise but in the execution (gameplay variability like shooter mechanics, main story). On top of that, 2016 was a bad year not only for DX but most AAA productions from different publishers as well. My personal guess is, SE thought it could leverage DX into sales ranks of a bigger mainstream audience, just like Capcom thought with Resident Evil. But these type of games do not scale that simply to a mass market appeal without losing their inherent DNA. That's a problem they could not solve and Square Enix like every stock listed publisher is looking for big sellers, not sophisticated reviews. At this time, Avengers is definitely bigger than DX.