Unidentified men on Friday set upon the building, bulldozing it to the ground.
According to members of the Nigerian high commission staff, some of the demolition team also threatened anybody who might challenge their actions.
The building was said to be located on a prime land disputed by the high commission and the traditional authority in the Ghanaian capital.
Protesters under the “Nigerian Lives Matter” banner complained about being openly discriminated against by Ghanaians and demand that the authorities investigate what they called a gross violation of their human rights.
The protesting Nigerians also accused Ghana of treating them with disdain for no discernible reason.
Meanwhile, the Nigerian government also summoned Ghana’s deputy ambassador to Nigeria to explain why the building was demolished.
The Ghanaian government has promised to investigate and expressed regret over the incident which has underlined sour relations between the two West African neighbours.
Their shaky relations came to a head during a so-called trade war last year after Nigeria shut down its borders and Ghanaians retaliated by forcing Nigerian shops in Ghana to close.