Hong Kong 1962, Chow Mo-Wan (Tony Leung Chiu-Wai), a journalist, rents a room from Mr. Koo. He will live there with his wife, a hotel receptionist. It's sheer coincidence that he moves in the same day that Su Lizhen (Maggie Cheung Man-Yuk) moves in next door, at Mrs Suen's place. Lizhen works as a secretary to Mr. Ho (Lai Chin), the boss of a shipping company. It's also a coincidence that both of them are moving in without help from their spouses. Chow's wife is working her shift at the hotel at the time of the move. Lizhen's husband, Mr Chan, is away on a business trip; he works for a Japanese company, and is often abroad. Despite having convivial and neighbourly landlords, Mr. Chow and Mrs. Chan often find themselves alone and lonely in their respective rooms.

Neither of them ever finds out how it began, but Mr. Chow and Mrs. Chan discover that their respective spouses are having an affair. The discovery shocks both of them. Chow, feeling hurt and wishing to understand how the affair happened, begins finding excuses to spend time with Mrs. Chan. They begin rehearsing what they will say to their spouses when they confront them with what they know. Then Mr. Chow invites Mrs. Chan to help him with a martial-arts series that he is writing for the newspaper. Their meetings are discreet, but people begin to notice. There seems no possibility that they, too, will drift into an affair. But Mrs. Chan's emotional reticence begins to haunt Mr. Chow and he finds his feelings changing. It's almost like being in love.

Four years later, as a Singapore-based reporter covering General De Gaulle's visit to Cambodia, Chow Mo-wan finds himself remembering an old story about a way of unburdening yourself of a secret you don't want anyone to know.