Web Services Performance, design considerations and ease of use

Employees at WSO2 published performance results of Axis2 vs. XFire along with the source code code used and the performance testing methodology. The Bileblogger criticized this with racial comments which I don't want to elaborate on here. This led to a series of nasty exchange of words. Shortly after this, Sun's JAX-WS benchmarked against Axis2. All this led to some constructive feedback and the question of usefulness and relevance of these benchmarks. Steve Laughran argues that one cannot realistically infer any result from these performance numbers. Anil John points out that these performance tests are not in par with the design considerations of web services. There's a point in what Anil says. Web services should simply be xml-in and xml-out in order to encourage interoperability and take advantage of powerful xml representation. Axis2 has in fact designed for the xml-in/xml-out model. This comes at the cost of ease of use, which is why data binding frameworks such ADB, JAXB, JiBX, etc have been introduced. We need to strike a balance between performance and ease-of-use. We need to come up with benchmarks that go well with design considerations (such as message based programming over RPC style, asynchronous web services, large data transfer, etc).