I found this 1853 map by J. F. Harrison through the New York Public Library Digital Collections that neatly captures the developmental extent of Manhattan at the time while also revealing the vestiges of the vanishing countryside.
I’ve circled the area in the East 50s where Cato Alexander’s Tavern had been until shortly before this map was drawn. You can still see the old country road—the Eastern Road, I think it was called—curving through the area and compare it to the street grid overlay that was just starting to fill in.
I believe that’s the main building, but all of the buildings were part of the property and more than one was used for serving guests. And yeah, they planed down a lot of hills and filled in a lot of valleys when they turned the countryside into Manhattan.
It would be nice to have a plaque on the spot, wouldn’t it?