Taking Back Sunday was formed in New York in 1999. The founder was a guitar player Eddie Reyes. Four of his friends joined the project right away. However, the permanent lineup was formed three years later. The group went through numerous member changes.
Eddie used to work with such groups as The Movielife and Mind Over Matter. Another member of the band Jesse Lacey performed with Rookie Lot. They were joined by a guitar player John Nolan, a vocalist Antonio Longo, and a drummer Steven DeJoseph. The latter soon left the band. Mark O’Connell replaced him. Longo left right after the group released its debut EP, Return Back Sunday. Two more band members left before 2001 was over. In February, Taking Back Sunday released a demo album, which featured five tracks, and embarked on a tour.
Before releasing the debut LP, Tell All Your Friends, the musicians recorded a popular music video for the song "Great Romances of the 20th Century". In 2000, the group signed a contract with Victory Records and started working on the debut album. The LP earned positive reviews from the fans but mixed opinions from the critics. It received gold status in the USA.
In 2004, the artists released a new LP, Where You Want to Be. They used new musical instruments to record the album, which made it truly different from the predecessor. The album was commercially successful. One of the singles, "A Decade Under the Influence", reached the 16th spot on the Alternative Songs Chart. The album sold over 650,000 copies. At that time, many organizers tried to book Taking Back Sunday for private events.