The Atari 5200 was developed during a period of time when the Atari 2600 was still the dominant video game console, yet Atari was beginning to feel the threat of competition in the form of the Colecovision and Mattel Intellivision. Developing a console with improved hardware was probably a good move for Atari, but the Atari 5200 suffered from a number of issues that prevented it from taking the world by storm.
The Atari 5200 was, as far as I know, the largest video game system ever released (not counting arcade cabinets and other things with built-in monitors and such), and suffered low-quality, poorly thought-out controllers. None of these problems helped sales.
The Atari 5200 was an Atari 8-bit computer in a fat sheep's clothing. Atari had a good plan when they started the project, yet they botched it with a series of odd decisions, the worst being the decision to use controllers that were analog (bad), not self-centering (worse), and broke with regularity. As a result, the 5200 is not nearly as widely collected as the other Atari systems. The decision to use the Atari 8-bit's custom chips also broke compatibility with the 2600. The need for an add-on to play 2600 games likely hurt Atari 5200 sales.
The Atari 5200 did have its advantages, of course. It had more memory (16KB) than even the Atari 7800 (4KB), and included the POKEY sound chip, while the 7800 used the same, sad sound chip the 2600 used. However, the 5200's hardware ultimately wasn't good enough to crush its competitors. It wasn't even good enough to displace it's successful predecessor, the Atari 2600.