Abstract: Wire densities of interconnection networks is a critical design consideration for Massively-Parallel Processing (MPP) supercomputers with tens of thousands of processing nodes. In these MPPs, high-bandwidth parallel buses are increasingly replaced by high performance and reliable serial links, which allow increased network connectivity i.e., multiple communication ports per processing node. Nevertheless these bit-serial links are affected by channel errors. The re-transmission costs on these comparatively low-bandwidth networks can have a significant impact on the performance of an application code, particularly for algorithms requiring frequent local communications. This research analyses the impact of communication buffer sizes and associated transmission protocols on off-node communication performance of an MPP system called QCDOC.