Comments
yourfanat wrote: I am using another tool for Oracle developers - dbForge Studio for Oracle. This IDE has lots of usefull features, among them: oracle designer, code competion and formatter, query builder, debugger, profiler, erxport/import, reports and many others. The latest version supports Oracle 12C. More information here.
Cloud Expo on Google News
SYS-CON.TV
Cloud Expo & Virtualization 2009 East
PLATINUM SPONSORS:
IBM
Smarter Business Solutions Through Dynamic Infrastructure
IBM
Smarter Insights: How the CIO Becomes a Hero Again
Microsoft
Windows Azure
GOLD SPONSORS:
Appsense
Why VDI?
CA
Maximizing the Business Value of Virtualization in Enterprise and Cloud Computing Environments
ExactTarget
Messaging in the Cloud - Email, SMS and Voice
Freedom OSS
Stairway to the Cloud
Sun
Sun's Incubation Platform: Helping Startups Serve the Enterprise
POWER PANELS:
Cloud Computing & Enterprise IT: Cost & Operational Benefits
How and Why is a Flexible IT Infrastructure the Key To the Future?
Click For 2008 West
Event Webcasts
Benchmark Bust-Up:The Middleware Company Responds
Benchmark Bust-Up:The Middleware Company Responds

(November 8, 2002) - As controversy swirled around the validity of the .NET vs J2EE benchmarking results posted this week by The Middleware Company, JDJ editor-in-chief Alan Williamson went straight to TMC for their side of the story.

JDJ: Did TMC approach Microsoft with respect to the production of this report? If so, did TMC approach BEA, IBM, Oracle, or SUN for similar participation in the production of the report?

TMC: Yes, we approached Microsoft. Our original intention for this benchmark was to put forth a more realistic comparison of J2EE and .NET to replace the original unfair PetStore application that Microsoft published. We challenged Microsoft to see if they were interested in a new comparison involving some recodes of the J2EE implementation and the .NET implementation. They were interested, and helped us by providing a testing lab and covering some of our costs.

We did not approach the J2EE vendors. The vendors' general position on tests such as these are that they do not want to get involved, because they want benchmarks to be performed by committees rather than by independent entities. If we do a round 2 benchmark, we will invite the J2EE vendors, so that they have the opportunity to get involved.

JDJ: You are quoted to having using the Sun's recommended best practices (blueprints) for the optimizations of the PetStore application. Why were the best practices as founded and published by TMC not employed?

TMC:The TMC team did work hard and made the J2EE system run 17x faster than the original PetStore application. However, after considering the community's feedback, we have concluded the system was not fully optimized for performance. There are important and necessary optimizations that were not performed. They were not performed primarily due to artificial time and money constraints on the project. However, we realize now that if we are to do any comparison of J2EE and .NET, then both must be shown at their finest.

We do believe the optimizations we did not have time to make would have made a material impact on performance. Because of this, if you are considering making a purchase decision involving J2EE and .NET, we encourage you to take a "wait and see" approach and look towards a second benchmark, which we are currently considering performing. Going forward, we are considering doing a second benchmark that takes the community's completely valid critique into account. Please stay tuned to TheServerSide.com for developments in this area.

JDJ: Did you enter into any contract with Microsoft for this? Were you legally obligated to publish your findings however they turned out?

TMC: Yes. The agreement was that regardless of who the victor was, the results would be published.

JDJ: Many J2EE vendors have commented that TMC didn't properly represent the J2EE community. When TMC engineers found the results to be going against J2EE, why wasn't outside assistance from the J2EE vendors sought?

TMC: The position of many J2EE vendors is that they do not want to get involved with independent benchmarks. They do not believe in the viability of PetStore as a useful benchmarkable application, and are spending their time and energy on committee-based benchmarks. This is a completely valid position for the vendors to take, and we respect that position.

JDJ: Sun has continually said that the PetStore application should never be used for performance testing, not even between J2EE app servers. Yet the PetStore was still used as the basis for this benchmark. What benefit did TMC see from continuing to use the PetStore application?

TMC: We used PetStore for two reasons:

1) We would have liked to port a real committee-based benchmark such as ECPerf to .NET, but that code is protected by Sun and we currently do not have permission to port it to .NET and perform a comparison.

2) Microsoft originally published a bogus report comparing a .NET PetStore with a J2EE PetStore, and showed .NET to be 10x faster. This was particularly embarrassing since, after all, they had used Sun's example. We wanted to debunk that comparison and do one more realistic -- for example, one that did not use stored procedures, which was in Microsoft's original comparison. So PetStore was the natural choice.

JDJ: We believe you are already thinking of revisiting the report with a new set of tests. Will Microsoft be involved? Have you invited the participation of BEA, IBM or any other J2EE vendor?

TMC: Yes, if we do a round 2, then we will invite the J2EE vendors to participate. Please stay tuned to TheServerSide.com for developments in this area.

JDJ: Speaking now on a personal level, what message would you like to give to the Java community who may be a little stunned at your findings?

TMC: Our advice is to not use this report when making purchase decisions, but rather, to either look towards committee-based benchmarks, or towards a round 2 PetStore bakeoff which we are considering doing right now.

About Alan Williamson
Alan Williamson is widely recognized as an early expert on Cloud Computing, he is Co-Founder of aw2.0 Ltd, a software company specializing in deploying software solutions within Cloud networks. Alan is a Sun Java Champion and creator of OpenBlueDragon (an open source Java CFML runtime engine). With many books, articles and speaking engagements under his belt, Alan likes to talk passionately about what can be done TODAY and not get caught up in the marketing hype of TOMORROW. Follow his blog, http://alan.blog-city.com/ or e-mail him at cloud(at)alanwilliamson.org.

In order to post a comment you need to be registered and logged in.

Register | Sign-in

Reader Feedback: Page 1 of 1

The only good to come from this fiasco is to show that when somebody asks how does A compare to B, the only correct answer is to do the comparison yourself, because you can't trust the so called experts to do it for you.

Do you go and trade your Chevey for a Honda when Honda wins a formulae 1 race? And then again, trade your Honda in for a Ferrarri when they win the next one? No. Why? Because your car does exactly what you need it to do at a cost much,much lower than the formulae 1 cars, or even the show-room Ferrarris.

Do you trade in your computers every 2 months because there's something faster on the market? No. Why? Because your computers are fast enough for what you use them for.

I'm a java programmer, but I don't give a rat's butt what the "final" results are. I'm developing very cost effective Java solutions on Linux machines for my clients. I've always been able to meet the performance targets required.

Remember, not everyone is Michael Schumiatcher nor are they developing the fastest Chess program on the planet. Millions of companies simply want to get a quality, cost-effective solution to a business problem.

When a client has Windows servers and is already using Visual Basic, I work with them to develop VB-based solutions. When they don't, I steer them towards a less expensive, but just as productive Java/Linux solution. If they insist on going the Microsoft route, so be it. All I, and we as IT professionals, can do, is provide enough information for them to make the best choice for thier company.

I've written only 1 small VB app in the last 2 years. Businesses out in the real world understand that IT is suppose to be about solving business problems, not about holy wars.

Fr what it's worth,
Don

TMC: "Our advice is to not use this report when making purchase decisions..."

Why did they bother going to all that effort when the result is useless? Many people have now expended lots of time discussing and debunking this "benchmark" - time that could have been put to more productive use.

TMC has no credibility left in this area and should be overlooked as a source of advice regarding .NET versus J2EE.

Question: did TMC approachthe J2EE vendors for participation in the production of the report?

November 8 TMC Answers: We did not approach the J2EE vendors.

This information should have been in the report.
This information should have been in your FAQ.

http://www.sys-con.com/java/article.cfm?id=1739

Hey guys,
The interview speaks by it self.
That starts as: "...we challenged MS"
and ends as: "... due to artificial time and money constraints".
All these years these guys are representing the "Java-bus" we take freely from a "Java-station".
Some day they "challenged" the "MS-bus" ... and left out of gas.
I find the message very clear and in the right way:
SHOW THE MONEY

The evil empire is at it again since Java and Linux is eating up its market share it is sweating a lot can you smell it..it stinks. Now with all its cash that it gets from MSOffice and desktop OS it is fighting for its life to have monopoly. VB has been a total disaster so now thier copy cat C# and .Not framework is a flop too. Well this should be easy just boycott MS products from your work and home and this company will disappear. We have switched to linux last year our expenses and Java dev time has been cut to half the time ..say bye bye to M$.

The Borg is definitely up to its old tricks, fresh from its buyout of the american justice system.

But with a twist this time around - such FUD can only mean one thing - extreme desperation.

Here are some tasty rebuttal factoids:

- embedded linux has the same market share, but higher growth rate, than M$ embedded.

How long before M$ embedded is toast in embedded market?? Embedded is >>>>>>> desktop.

Linux has higher growth than M$ in server market. How is M$ going to stop Linux?

If M$ cannot stop Linux in server and embedded, how long before the penguin eats desktop?

- Mindcraft played similar games a while back. How come no more mindcraft potshots at Linux?

All mindcraft did is open the door wider for Linux, and, I rather suspect, that the bounce of this little maneuver will strengthen Java too.

It is starting to look like Big Bad Bill$ is his own worst enemy.

How about another license fee hike for Office????

Make My Day....

A couple of points:

1. Who really trusts benchmarks? Anyone with a bias can find or generate benchmarks to support their claims. They are, perhaps, food for thought and further investigation cannot be fully trusted when used as a marketing tool. Is Microsoft the only vendor that "buys" benchmarks? Hardly... What it comes down to is that I don't trust ANYONE's benchmarks but my own :)

2. This J2EE / .NET feud is just silly. Realistically, each is strong in places where the other is lacking... there is no "perfect" choice here - only a choice that is perfectly suited to your environment, budget, skills, and ideology. We dropped Java in favor of .NET this past year because for us it made the most sense: licensing and maintenance for our Java tools was astronomical, our development targets Windows exclusively, and we really missed the rich UI we were used to in native Windows apps. Is that the choice everyone should make? Absolutely not! If you are happy with what you use, *be* happy... but don't disparage other choices.

It just seems that this turns into a case of people wanting to bash anything Microsoft does, even when they do something good!

I think the next benchmark for both .Net and J2EE should be run on Linux boxex -- I suspect J2EE will be infinitely faster... ;^)

I haven't see such a fast backpedal since my bicycle brakes went out going down "Suicide" hill when I was 10.

This benchmarking comparison was advertised as a fair comparison between best practice implementations of .NET and J2EE. Now TMC themselves are stating that the results of this benchmark are not to be trusted?

This from a company that is supposed to be experts at implementing and mentoring enterprise J2EE implementations?

The irony is so deep here that I can't even conceptualize an effective metaphor to compare it to. Wow.

Java community - let's not lose sight of the real issue here - Microsoft bought and paid for this benchmark. TMC is going to pay for this charade with a massive loss of credibility. Just don't forget that MS has not stopped in it's efforts to undermine the Java industry, and although we may be able to see through their marketing hype, there are plenty of uneducated decision makers out there that are reacting to marketing fluff such as this.

Excerpt 1: "Microsoft originally published a bogus report comparing a .NET PetStore with a J2EE PetStore, and showed .NET to be 10x faster."

Excerpt 2: "The TMC team did work hard and made the J2EE system run 17x faster than the original PetStore application."

The question: why TMC J2EE solution is still slower?


Your Feedback
Greg Turner wrote: The only good to come from this fiasco is to show that when somebody asks how does A compare to B, the only correct answer is to do the comparison yourself, because you can't trust the so called experts to do it for you.
Don wrote: Do you go and trade your Chevey for a Honda when Honda wins a formulae 1 race? And then again, trade your Honda in for a Ferrarri when they win the next one? No. Why? Because your car does exactly what you need it to do at a cost much,much lower than the formulae 1 cars, or even the show-room Ferrarris. Do you trade in your computers every 2 months because there's something faster on the market? No. Why? Because your computers are fast enough for what you use them for. I'm a java programmer, but I don't give a rat's butt what the "final" results are. I'm developing very cost effective Java solutions on Linux machines for my clients. I've always been able to meet the performance targets required. Remember, not everyone is Michael Schumiatcher nor are they developing the fastest Chess program on the planet. Millions of companies simply want to get a quality, cost-effective...
Steve wrote: TMC: "Our advice is to not use this report when making purchase decisions..." Why did they bother going to all that effort when the result is useless? Many people have now expended lots of time discussing and debunking this "benchmark" - time that could have been put to more productive use. TMC has no credibility left in this area and should be overlooked as a source of advice regarding .NET versus J2EE.
Dion Muddle wrote: Question: did TMC approachthe J2EE vendors for participation in the production of the report? November 8 TMC Answers: We did not approach the J2EE vendors. This information should have been in the report. This information should have been in your FAQ. http://www.sys-con.com/java/article.cfm?id=1739
Costas Ginos wrote: Hey guys, The interview speaks by it self. That starts as: "...we challenged MS" and ends as: "... due to artificial time and money constraints". All these years these guys are representing the "Java-bus" we take freely from a "Java-station". Some day they "challenged" the "MS-bus" ... and left out of gas. I find the message very clear and in the right way: SHOW THE MONEY
Jim wrote: The evil empire is at it again since Java and Linux is eating up its market share it is sweating a lot can you smell it..it stinks. Now with all its cash that it gets from MSOffice and desktop OS it is fighting for its life to have monopoly. VB has been a total disaster so now thier copy cat C# and .Not framework is a flop too. Well this should be easy just boycott MS products from your work and home and this company will disappear. We have switched to linux last year our expenses and Java dev time has been cut to half the time ..say bye bye to M$.
Mark Tompkins wrote: The Borg is definitely up to its old tricks, fresh from its buyout of the american justice system. But with a twist this time around - such FUD can only mean one thing - extreme desperation. Here are some tasty rebuttal factoids: - embedded linux has the same market share, but higher growth rate, than M$ embedded. How long before M$ embedded is toast in embedded market?? Embedded is >>>>>>> desktop. Linux has higher growth than M$ in server market. How is M$ going to stop Linux? If M$ cannot stop Linux in server and embedded, how long before the penguin eats desktop? - Mindcraft played similar games a while back. How come no more mindcraft potshots at Linux? All mindcraft did is open the door wider for Linux, and, I rather suspect, that the bounce of this little maneuver will strengthen Java too. It is starting to look like Big Bad Bill$ is his own worst enem...
Tim Ferrell wrote: A couple of points: 1. Who really trusts benchmarks? Anyone with a bias can find or generate benchmarks to support their claims. They are, perhaps, food for thought and further investigation cannot be fully trusted when used as a marketing tool. Is Microsoft the only vendor that "buys" benchmarks? Hardly... What it comes down to is that I don't trust ANYONE's benchmarks but my own :) 2. This J2EE / .NET feud is just silly. Realistically, each is strong in places where the other is lacking... there is no "perfect" choice here - only a choice that is perfectly suited to your environment, budget, skills, and ideology. We dropped Java in favor of .NET this past year because for us it made the most sense: licensing and maintenance for our Java tools was astronomical, our development targets Windows exclusively, and we really missed the rich UI we were used to in native Windows apps. I...
W. Nathaniel Mills, III wrote: I think the next benchmark for both .Net and J2EE should be run on Linux boxex -- I suspect J2EE will be infinitely faster... ;^)
Jonathan House wrote: I haven't see such a fast backpedal since my bicycle brakes went out going down "Suicide" hill when I was 10. This benchmarking comparison was advertised as a fair comparison between best practice implementations of .NET and J2EE. Now TMC themselves are stating that the results of this benchmark are not to be trusted? This from a company that is supposed to be experts at implementing and mentoring enterprise J2EE implementations? The irony is so deep here that I can't even conceptualize an effective metaphor to compare it to. Wow. Java community - let's not lose sight of the real issue here - Microsoft bought and paid for this benchmark. TMC is going to pay for this charade with a massive loss of credibility. Just don't forget that MS has not stopped in it's efforts to undermine the Java industry, and although we may be able to see through their marketing hype, there are plen...
Vlad Kosulin wrote: Excerpt 1: "Microsoft originally published a bogus report comparing a .NET PetStore with a J2EE PetStore, and showed .NET to be 10x faster." Excerpt 2: "The TMC team did work hard and made the J2EE system run 17x faster than the original PetStore application." The question: why TMC J2EE solution is still slower?
Latest Cloud Developer Stories
All in Mobile is a mobile app agency that helps enterprise companies and next generation startups build the future of digital. We offer mobile development and design for smartphones, tablets and wearables. Our projects cover the latest and most innovative technologies - voice ass...
NanoVMs is the only production ready unikernel infrastructure solution on the market today. Unikernels prevent server intrusions by isolating applications to one virtual machine with no users, no shells and no way to run other programs on them. Unikernels run faster and are light...
The dream is universal: heuristic driven, global business operations without interruption so that nobody has to wake up at 4am to solve a problem. Building upon Nutanix Acropolis software defined storage, virtualization, and networking platform, Mark will demonstrate business lif...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO Silicon Valley 2019 will cover all of these tools, with the most comprehensive program and with 222 rockstar speakers throughout our industry presenting 22 Keynotes and General Sessions, 250 Breakout Sessions along 10 Tracks, as well as our ...
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, vi...
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
Click to Add our RSS Feeds to the Service of Your Choice:
Google Reader or Homepage Add to My Yahoo! Subscribe with Bloglines Subscribe in NewsGator Online
myFeedster Add to My AOL Subscribe in Rojo Add 'Hugg' to Newsburst from CNET News.com Kinja Digest View Additional SYS-CON Feeds
Publish Your Article! Please send it to editorial(at)sys-con.com!

Advertise on this site! Contact advertising(at)sys-con.com! 201 802-3021



SYS-CON Featured Whitepapers
Most Read This Week
ADS BY GOOGLE