The most obvious means of interaction, and perhaps the most successful, is the introduction of mathematics into literature. Certainly there is a reasonable body of mathematical science fiction. The classic collection The Mathematical Magpie and Rudy Rucker's more recent anthology Mathenauts gather excellent examples of this genre. Indeed, much of Rucker's work is informed by his professional knowledge of science in general and mathematics in particular. Kim Stanley Robinson's short story, "The Blind Geometer," is a beautiful story which manages to evoke much of the elegance of mathematics. Mathematics has also inspired a moderate amount of poetry, even if much of it is low-brow.
There has been a certain amount of work towards introducing literary techniques into the analysis of mathematics. The success of these endeavors is less convincing. It's not entirely clear why this might be a good idea; but as literature and mathematics compete for discursive space, it's not surprising to see such moves. As an example, one might care to peruse the work of Gaston Bachelard. He strove to demonstrate that mathematics is but an expression of difference, just like any other (literary) language. If he could win this point, then mathematics would be laid bear to the tools of the literary critic.
In what may be regarded as a textbook case of "if you can't beat em, join em," Bachelard has been joined by a host of others in attempting to use the tools of mathematics to explain literature. Mathematics is a place where our society apparently finds truth. So it's not surprising that practicioners of all sorts of (un)disciplines try to borrow the established ethos of the queen of the sciences. Often, this takes the form of vigorous assertions of parallels. Bachelard affords us an example:
We shall need real courage if we are to found projective poetry before there is ever metrical poetry, just as sheer genius was needed to discover, very late in the day, that beneath metrical geometry there lay projective geometry, which is in fact essential and primitive. Poetry and geometry are completely parallel here. The basic theorem of projective geometry is as follows: what elements of a geometric form can, with impunity, be deformed in aprojection in such a way that geometric coherence remains? The basic theorem of projective poetry is as follows: what elements of a poetic form can, with impunity, be deformed by metaphor in such a way that poetic coherence remains? In other words, what are the limits of formal causality? [101]
It's hard to know where to initiate a critique of this particular passage. Items which come to mind include the mischaracterization of projective geometry in general, and of the relationship between projective geometry and other geometries in general; the difference between definition, axiom, theorem and question; and the assertion of theorems which are in no way provable. Even if the ideas of mathematics provide a metaphor or impetus for inquiry, one can't go around proving theorems about literature in any sort of mathematical sense. (This may be a good point to discuss positivisim, but it's a topic I'm loathe to take up right now.)
Still, at least Bachelard's exposition is lucid. Centered around the word topology, for example, one can find any number of mathematically inane statements, such as those by Julia Kristeva and Kelly Cherry. These are the statements which form the motivating principle of this web.