Anyone else think an open beta is a bad idea?

Pages : [1] 2 3 4

Imposter
05-13-2003, 10:44
Ok call me crazy, but I don't want to see an open beta. Here's why-

-The main goal of an open beta vs a closed one is that more people theoretically means more bugs would be discovered. But I think it will not scale that way. You'll get the same bug reported more times.

-You'll get an absolute avalanche of bug reports. Most will be of the same bug but you'll have to categorize them and somehow weed out the duplicates. Theres no way to do that with a computer, it will have to be by hand. You'll probably need someone who understands the code to help categorize them so you know where to start searching for a bug...that takes up a lot of resources.

-I guess a problem I have with it is that an open beta takes away from the surprise of playing a new game. Everyone will want to play a free beta. They'll use that to decide if they want to buy the game. But they'll be playing a crappy, unfinished version of the game. I think you'll lose purchasers of the game because of it.

-It's basically a free trial of the game. So it could generate buzz, but like I said above, buzz isn't always good. See T2.


So what are other options?

-A small closed beta. Downside is you won't find as many errors. Upside is the "surprise" factor of a new game stays there when people first play the game.

-A large (1000+) closed beta. You'll find more bugs but you'll also get leaks where people talk on forums (like this) saying what they think of the game. There's no real way to avoid that. T2 was like this. But for all the errors it found they weren't fixed, which is actually a good point that a beta is only as good as how much time the dev team has to respond. If they waste time with other things instead of fixing bugs then a beta isn't worthwhile.

-Hire a good Q&A team. Find more bugs, get better results, work faster. May miss more obscure hardware related bugs, but that should be rare if the Q&A team is good. Didn't thrax used to do q&a for sierra? I may be mistaken.

So it becomes a cost/benefit analysis. Is paying a good Q&A team more cost effective than a large beta? I think so but I'm obviously not in the industry and am just basing it on intuition. It seems whenever there is an open beta people get tired of the game or talk about it's bugs and don't end up buying the game, ie: Planetside.

Sure players want a free preview of a game before they purchase it. But that can be covered by a seemingly out-dated idea, the demo. Give them 2-3 levels of the single player in a demo and let that generate buzz.

Anyway, I eagerly await the open beta so I can see how the game plays ;)

vempire
05-13-2003, 11:10
You do make some good points, but here is something else to think about.

In a closed beta, you will probably be finding a more select group of people. More then likely you would see the biggest names in the Tribes community getting preference for being admitted into the closed beta. After that you would probably see that most of the other people in the beta would be experienced tribes players. This would be good in that they would know how the games have worked inside and out in the past. The problem with that though would be bias and prior knowledge. In any kindof QA/testing environment it is a good idea to get a mixture of people who know what they are doing as well as people who don't. People who know what they are doing will do everything right, which is good because they will find certain bugs. People who don't know what they are doing are also good because they will find other bugs by doing wierd combinations of things.

I worked in software/hardware QA/testing for about a year at the company I work at right now, so I'm quite aware of how to find bugs. We would always do the bulk of the testing on our machines with people who knew how to work them, then we would let random employees here use the machines. You would be amazed at how many issues came up through people doing the wrong things. Then we would go back and kill those bugs. In the end, it makes for a far tighter end product.

Imposter
05-13-2003, 11:16
I think you're right in that a lot of "big names" would make it into a small closed beta like in T2. And as you say that has advantages and disadvantages. However they could take that into consideration when they take people into the test.

I'd rather see them hire a Q&A test and do no public beta test.

Void|deadjawa
05-13-2003, 11:29
I think large betas 20000+ that blizzard uses now are the perfect number. That way you get a lot of varied systems played on, but its still manageable.

Imposter
05-13-2003, 11:33
How is 20000 players managable? How big is blizzard compared to Irrational I wonder? You have to spend a good bit of resources to deal with the input from a large beta.

Kendo
05-13-2003, 11:43
Well cant they just put in like one map for the beta or somthing so people arent totally exposed to the entire game and they will still wanna see what all the other stuff is like, and buy it.

As long as essential things are put in the beta that will test as much of the coding as possible without ruining the surprise would be good to me.

Imposter
05-13-2003, 11:49
If they have a beta with just one map, and leave out things to still keep that "surprise" factor, then they would have missed the entire point of a beta test, because they wouldn't have tested that part of the game they left to be a "surprise".

jsut
05-13-2003, 11:53
They said there would be a very small closed beta before the open beta. "more of an alpha test" was i think how they refered to it.

Imposter
05-13-2003, 11:56
T2 had a beta test, that turned into the meta test.

I'm just saying I wouldn't mind if they just hired a q&a team to do the testing and we only got to play the game when it went on sale.

LouCypher
05-13-2003, 11:58
I would hope Sierra and KP both learned from TR2 that closed betas aren't a very good idea.

Imposter
05-13-2003, 11:59
The "closed" beta of t2 has over 1000 people right? That would fit into my large beta test thing above. It wasn't that they didn't report the bugs, its that the bugs weren't fixed. Having 10000 wouldn't cure that problem.

jsut
05-13-2003, 12:00
From you're own Q

[19:38] <Colosus> *InvalidName* While there has been discussion about having an open beta, will there still be offical beta sign ups for a closed beta as well?
[19:39] <KineticPoet> Maybe not. No, the whole point of an open beta is that its open...there'll be no signups for a closed beta
[19:39] <Thrax> We will have a closed beta before the open beta. It probably won't involve signing up. It will be very small.
[19:40] <Thrax> More like an Alpha test.
[19:40] <Thrax> The true beta will be very open. As open as we can make it.

[4M]TheReverend
05-13-2003, 12:01
/agree Vempire

I am looking forward to beta...as long as you don't have to DL it from FilePlanet...

Imposter
05-13-2003, 12:01
I know what they're plans are man.

Imposter
05-13-2003, 12:02
Why would you want to play in a beta? I'd rather have a demo to test the game with a few weeks before the game comes out. A beta is a demo with bugs.

LouCypher
05-13-2003, 12:17
TheReverend']/agree Vempire

I am looking forward to beta...as long as you don't have to DL it from FilePlanet...
AMEN!

My preference would be something like BitTorrent.

SarcaStick
05-13-2003, 12:20
I'll **** Thrax if I don't get beta!

jsut
05-13-2003, 12:27
I'd rather play in a beta with bugs, than a release version with bugs. I'm sure they could accomplish a non-buggy version of the game with a closed beta, or with hardcore QA, but nothing is going to appease the community until they get to play it.

By having an open beta, assuming the game is stable, and balanced by the end of said beta, sales on release will be better. How many people on TW aren't going to buy the game right away until they get the TW stamp of approval. How much better will initial sales be if that stamp of approval comes during beta, instead of a day or two after release? How much higher will the pre-order's be?

Plus, we have the whole T2 issue to deal with too. How many people on TW think T2 + closed beta = sucks => T3 + closed beta => sucks. An open beta of T3 disassociates the game with T2 even more by not stifling discussion of the game before release. If the game is stable and balanced and fun, nothing could be a better marketing tool than that. If the game sucks, you save your $50, and the game dies a horrible death. However, if the game sucks and there was a closed beta, you might spend you're $50 and be pissed off, and the game will still die a horrible death.

The only thing a demo represents to me is a delay in release. It's another thing to work on. If they do a demo it should be released the day the game goes gold, or close afterwords. I don't think a beta should mean that their is no demo, because beta's tend to not work after release, where as a demo does, and there will be a lot of people that didn't try the beta that would want to play a demo.

Personally I could care less if the beta is open or not as long as i'm on it. It being open means one thing to me, that i don't have to worry about whether or not i'm going to be on it.

Imposter
05-13-2003, 12:55
But how many games are actually stable at the end of a beta test? Most seem to leave bugs in to be fixed before shipping...and I have NEVER seen a beta test generate positive news for a game. Take warcraft 3. If there had not been a beta of that game I bet it would have had MUCH higher sales. Instead people played it, didn't like it, and told everyone so. That doesn't mean its not a fun game, it means that some people didn't think it lived up to the hype.

Seriously, an open beta never works for the company that I've seen. I could certainly be wrong though, perhaps someone could provide a good counter-example.

Menzo
05-13-2003, 13:23
Take warcraft 3. If there had not been a beta of that game I bet it would have had MUCH higher sales. Instead people played it, didn't like it, and told everyone so.

I hate to tell you this, but Warcraft 3 has sold like gangbusters. They shipped over 1.5 million copies to retail on day one alone.

Once you achieve Warcraft 3-style success there's no way you can suggest that they could have sold more if they had done things differently, because there are only three or four PC games ever that have done as well (and most seem to be Blizzard games).

By all accounts, the open beta for Warcraft 3 was a great success - and if we can be half as successful as they were we'll assure the life of Tribes for a long, long time.