When you finally decide to stir from your recumbent position on one of the wonderful beaches on Key West, or put down your mixed drink as you barhop on Duval Street, you’ll find that this island city offers more than just sun, sand, key lime everything, and the haunts of Ernest Hemingway. There’s a noteworthy built environment, too. Most houses are made of wood (highly susceptible to hurricanes, which have miraculously spared this outlier for many years now), but plenty of more substantial buildings add another dimension to this supremely laid-back locale. These are my favorites.

#1 Old Post Office and Custom House

One of the largest buildings in the historic district of Key West, the beautiful old post office and custom house joined the National Register of Historic Places in 1973, and it’s easy to see why. Opened in 1891, this four-story red-brick Richardsonian Romanesque beauty features broad arches along a wraparound porch. The building saw a long litany of legal cases, from rum running to the investigation into the sinking of the USS Maine in 1898, before it was transferred to the U.S. Navy in 1932. After the Navy departed, the building remained abandoned for nearly two decades. Today, the quondam post office and custom house is home to the Key West Museum of Art & History, with solid exhibits about the history of Key West, and of Henry Flagler’s ill-fated railway that was demolished during the devastating 1935 Labor Day hurricane, permanently ending rail service to the Keys.

#2 Basilica of St. Mary Star of the Sea

On a particularly sudorific day, just a few blocks off Duval Street in a quiet neighborhood, I took a seat in the Basilica of St. Mary Star of the Sea and enjoyed the refreshing cross-ventilation provided by a series of open doors below simple stained-glass windows along both side aisles. One of the oldest Catholic parishes in Florida, this church, completed in 1905 and light on decorative details, features twin silver spires; a white and powder blue (the color often associated with Mary) airy interior with slender columns supporting arches with soffits that resemble waffles; and a grotto outside with an abundance of candles. Above the altar, the only stained-glass window with a figure, St. Mary, lets in a kaleidoscope of colorful light. Added to the National Register of Historic Places in 1971, the church was elevated to the status of minor basilica by Pope Benedict XVI in 2012. When you’re done appreciating this church and its peacefulness, you can purchase a memento from the only Catholic gift shop within 100 miles.

#3 St. Paul’s Episcopal Church

Another major Key West house of worship, St. Paul’s Episcopal Church stands right in the midst of the action on Duval Street. It’s a crisp white structure with a graceful corner bell tower and red entry doors typical of an Episcopal church. Inside, I admired the intricate wooden ceiling and the highly detailed stained-glass windows. The fourth version on this site (two previous churches were destroyed in hurricanes, a third by fire), the current building dates back to 1912. Of course, with a history of more than 100 years, there’s bound to be a ghost story or two, and St. Paul’s is reputedly haunted by the ghosts of one of the island’s founders as well as a sea captain, a man obsessed with driving pirates out of Key West, and a handful of children who died in a fire when a former pastor, in a jealous rage over his wife’s infidelities with the deacon, set fire to the church without knowing they were inside. That myth has been debunked, however: The fire was deliberately set about a block away by the Spanish Empire in an effort to take out Key West.

#4 Southernmost House in the United States

At the southern end of Duval Street, a brick Queen Anne Victorian-style home in pastel pinks and blues that was built as a one-bedroom house (despite its very large size) in 1896 for a judge bears a unique distinction: it’s the southernmost house in the United States. It’s also a popular site, standing only a block from the iconic buoy-shaped marker indicating the southernmost point in the United States. With its porches, round tower, peaked roofs, widow’s walk, and tiled risers on the entrance staircase, I imagined that Judge Vining Harris must have really loved his home, especially because it cost him a quarter of a million dollars to build. Over the years, the house was converted into a speakeasy club during Prohibition and then, in 1939, into a Cuban nightclub that operated for 15 years. Five U.S. presidents have stayed here throughout its history, and you can, too: It’s now a luxury 18-room hotel.

#5 Old First National Bank

On the opposite end of Duval Street, about a mile from the Southernmost House, the exotic triangular building that originally served as the First National Bank anchors this end of the famous strip. Started with capital from local cigar manufacturers, the bank constructed this impressive Moorish Revival–style edifice in 1897. It sports a wonderful columned balcony with a fine balustrade and brackets, a red-tile roof on a corner turret, and eye-catching patterns of polychromatic brickwork, with red and cream-colored bricks forming horizontal stripes, emphasizing lintels and arches over windows, and creating diamond patterns toward the top. The bank earned a random tidbit of history when its president refused to cash a $1,000 royalty check issued to Ernest Hemingway for A Farewell to Arms.

Five Runners-up

• Strand Theater (1920s)
• E.H. Gato Cigar Factory (1920)
• San Carlos Institute (1890)
• Old City Hall (1876)
• Oldest school in Key West (1847)