"That's When" received generally positive reviews from critics, with several of whom praised Swift's and Urban's collaboration and the songwriting. Saloni Gajjar of
The A.V. Club wrote that the collaboration "hits all the right soothing notes". Kelsey Barnes from
Gigwise wrote that Swift's collaboration with Urban "fit perfectly alongside her other collaborations". Heather Taylor-Singh of
Exclaim! selected "That's When" as exemplary of the "From the Vault" tracks as "a fun treat for fans who can't get enough of Swift's vivid storytelling". Joseph Hudak of
Rolling Stone called "That's When" an "introspective retelling of a romance at the brink". Olivier considered it an album highlight and a catchy duet that gives a glimpse into Swift's early career in country music. Hannah Mylrea of
NME also viewed it as a highlight from the album and said it was a "woozy" song containing a "layer of slick
1989-style production" and "euphoric layered vocals". Erin Browne from
Vulture commended how Antonoff kept the song "
twangy and country", which he thought those aspects made "That's When" faithful to the music of the original
Fearless while also being equal to the album's re-recording. Others were more reserved in their praise. Jackson Langford of
MTV ranked "That's When" fifth of the six "From the Vault" tracks and praised Urban's presence on the track, but believed that the song "still falls just short of greatness" and that it would have been better with Swift's early southern twang.
The Observer writer
Kitty Empire considered "That's When" one of the "mixed bag[s]" of the album's "From the Vault" tracks. Chris Willman of
Variety wrote that while "it's lovely to hear [Swift and Urban] together", the song does not feel as immersive in comparison to the other songs that made it into the original album, and he dubbed the track and the chords as "a slightly more balladic version" of the fellow album track "
You Belong with Me" (2009), which he deemed superior to it. In a similar sentiment,
Jon Pareles of
The New York Times commented that Swift left the song out of the album due to the chorus having a similar melody to "You Belong With Me", which he believed that it "perfectly" utilized. Later reviews of "That's When" were featured in lists and rankings of Swift's discography.
Rolling Stone Rob Sheffield and
Vulture Nate Jones ranked it at 234 and 224, respectively, in worst-to-best lists of Swift's catalog. Sheffield wrote that the song's theme of forgiving someone for breaking their heart without reason was in the style of her tracks "Afterglow" (2019) and "
Back to December" (2010). Josh Kurp from
Uproxx felt that "That's When" was a "fine song that fails to leave much of an impression", while Jones believed that it was similar to "You Belong with Me" and that Urban added "a bit of country
verisimilitude but not much interest". ==Credits and personnel==