So says one excavator, in the New York Times:

The story goes back to 1976 when lobster divers first found potsherds studded with barnacles just off Governor's Island in the bay. Then a Brazilian diver brought up two complete jars with twin handles, tapering at the bottom, the kind that ancient Mediterranean peoples widely used for storage and are known as amphoras. Brazilian experts disagree over the age of the jars, which have been turned over to the Navy and stored them in a warehouse.Mr. Marx, who has long sought to prove that other sailors reached the Americas well before Columbus, obtained permission to explore the site in late 1982. Diving at a depth of about 90 feet, he found the parts of perhaps 200 broken amphoras and several complete ones, he said in an earlier telephone interview.According to Elizabeth Will, a professor of classics and specialist in ancient Roman amphoras at the University of Massachusetts at Amherst, the jars are very similar to the ones produced at Kouass, a Roman Empire colony that was a center for amphora-making on the Atlantic coast of Morocco.

It's kind of a funny story, as some rich guy claims the amphorae are his which he had made as replicas before dropping them off in the water to make them look older.  I know of no other evidence which remotely suggests the Romans made their way across the Atlantic.