Recent direct imaging discoveries suggest a new class of massive, distant planets around A stars. These widely separated giants have been interpreted as signs of planet formation driven by gravitational instability, but the viability of this mechanism is not clear-cut. In this paper, we first discuss the local requirements for fragmentation and the initial fragment mass scales. We then consider whether the fragment's subsequent growth can be terminated within the planetary mass regime. Finally, we place disks in the larger context of star formation and disk evolution models. We find that in order for gravitational instability to produce planets, disks must be atypically cold in order to reduce the initial fragment mass. In addition, fragmentation must occur during a narrow window of disk evolution, after infall has mostly ceased, but while the disk is still sufficiently massive to undergo gravitational instability. Under more typical conditions, disk-born objects will likely grow well above the deuterium burning planetary mass limit. We conclude that if planets are formed by gravitational instability, they must be the low-mass tail of the distribution of disk-born companions. To validate this theory, ongoing direct imaging surveys must find a greater abundance of brown dwarf and M-star companions to A stars. Their absence would suggest planet formation by a different mechanism such as core accretion, which is consistent with the debris disks detected in these systems.