"Camulodunum" at firstsite Colchester. 

Colchester today is rough around the edges, a garrison town in Essex with an air of having fallen on hard times. Boarded-up shops and a proliferation of pawnbrokers tell a melancholy tale.

But once upon a time this was Camulodunum, Britain’s oldest recorded town. Emperor Claudius arrived in 43 AD with an invasion force complete with the latest shock-and-awe weapons of the day, in the form of armoured elephants. He made the town his capital, so this corner of thoroughly modern Essex is also shot through with signs and symbols of a classical past: fragments of a temple poking up out of parkland; an ancient gateway hard by a four-lane underpass; and – ensconced behind pigeon-netting high on the front of the town hall – a statue of the local heroine, Boadicea, who once led the ancient Britons in revolt against their Roman rulers.

The latest addition to the civic fabric of the town is a brand new £25 million arts centre: a large, low, curvilinear building entirely clad in bright gold copper-aluminium panels, designed by a New York-based Uruguayan architect called Rafael Vinoly. Its official name is Firstsite, although Vinoly himself has christened it "The Golden Banana". Its opponents argue that whatever the colour it is no more than a white elephant, every bit as unwelcome as those that invaded back in the days of Claudius. Firstsite has been much complained about locally, which is perhaps inevitable in a time of recession.

The building is certainly striking, and certainly like nothing else in Colchester. It resembles a huge spaceship but also looks like a fragment, a piece of some mysterious larger whole. It might almost be a section of Frank Gehry’s Guggenheim Bilbao, broken off and turned from silver to gilt. It also evokes the visor of a Roman legionary’s helmet, in which case some form of apt allusion to Colchester’s past may be inferred. At once light-filled and physically light, it has been constructed on extremely shallow foundations. It stands, almost gingerly, on a Scheduled Ancient Monument, namely the remains of an ancient Roman villa complex, visible most notably in the form of a delicate mosaic decorated with dolphins and other sea creatures, displayed beneath glass at the centre of the building. Oyster shells were found at the site of the original dig, suggesting that Colchester Natives were as popular in Roman Britain as they are today.

Within the contours of the architectural banana, space has been found for a decent-sized auditorium and cinema as well as a cafe and restauarant with a magnificent, grittily urban view of the ever-moving transport hub that is the city’s bus centre. For some reason the local council has plans to move the buses elsewhere and landscape the tarmac into something more conventionally picturesque. That would be a shame and might well strengthen the feelings of those who feel that Firstsite is not really for them – not a building that wants to be part of the town, but one that wants to talk down to it from a patronising height.

In truth, that does not seem to be the case. Extremely generous provision has been made for the Education Department which has been given a strong brief to draw local children into subjects ranging from contemporary art to ancient archaeology; as well as two large and welcoming spaces in which to do so.

The opening exhibition, entitled "Camulodonum", is an approachable and playful charting of pathways between Colchester’s history and the traditions of modern and contemporary art. Slices of the town’s past, such as photographs of the Colchester Pageant of 1909 – an almost militarily efficient restaging of episodes from the town’s history, with a cast of thousands directed by a splendidly named pageant-master named Louis Napoleon Parker – are juxtaposed with tenuously Colchester-apt works of art such as Andy Warhol’s screenprint of 1969, Soup II: Oyster Stew. Elsewhere curator Michelle Cotton has successfully unearthed a rich vein of art that worries away at the themes of archaeological finding, the relations between past and present. Contemporary Vietnamese artist Dahn Vo’s large installation is a sinister colossus for the twenty-first century, formed from fragments of a scale copy of the Statue of Liberty – as if to imagine some world of the future where these remains might be all that survives of a city once called New York. Bill Woodrow’s Car Door, Boot and Wing with Roman Helmet, of 1982, is itself a relic cut from the scrapyards of Thatcher’s Britain, an object formed from two old car doors, eviscerated into long strands of sheet metal, which have in turn been shaped into a mock-up of a Roman legionary’s helmet. It looks like a joke on the future phrased as a sculpture: what would people in the year 3012 make of us now, if they were to dig this up?

More to the immediate point, what will the people of Colchester make of Firstsite? The first signs are encouraging. Perhaps, with the passage of time, even the building’s most vocal opponents will soften to it. For better or worse, the town of oysters is now also a town of contemporary art.