Harvard graduate student Andrew Hall investigates this question in a new paper (PDF) that makes use of an innovative regression discontinuity model. Basically, he uses estimates of the ideological moderation/extremism of candidates for U.S. House primary elections and looks to see how the party does in the general election when it nominates a more extreme candidate. The regression discontinuity model looks like the graph below, where the horizontal axis shows the primary victory margin of an extremist and the vertical axis shows the party's general election vote share.
When an extremist (as measured by primary-election campaign receipt patterns) wins a “coin-flip” election over a moderate, the party’s general-election vote share decreases by approximately 12 percentage points, and the probability that the party wins the seat decreases by 38–46 percentage points. This electoral penalty is so large that randomly nominating the more extreme primary candidate causes the district’s subsequent roll-call representation to move away from the party’s ideology, in competitive districts. In safe districts, however, extremists win the general election often enough to cancel out these roll-call reversals. The tradeoff that primary voters face between supporting “electable” candidates and supporting more ideological candidates thus varies across district types.(h/t Eric McGhee)