Preface

This coming November (2018) the Mining and Smithing beta is slated to be released. Jagex has a checkered past when it comes to beta launches. The first widespread beta was RuneScape 2. Other betas have included: RuneScape HD, EoC, HTML5, RuneScape3, and finally mobile. These betas were largely carried out to gain community support. What should the goal be of a beta for both Jagex and the player base?

The Elephant in the Room: The EoC Beta

• The EoC beta allowed players to use any type of gear at any level. As a result most just tested top tier gear (tier 80 at the time) because they didn’t have it in game.
• Opposition to a change in the combat system was loud, so much so that players with constructive feedback felt drowned out.
• A reluctance to participate in the beta by segments of the community who didn’t like what was on offer. This resulted in a valuable portion of community feedback being missed entirely.
• Fast forward 2013: Old School RuneScape and later legacy mode in RS3.

What can be done to prevent another “EoC” style beta?

• Should beta testing be closed for the first 80% or until the beta is deemed feature complete?
• Should testers be forced to sign an NDA?
• Clear outcomes for each phase of testing
• Don’t do game design by opinion poll
• Dedicated feedback channels that can be private
• ???

A RuneScape Test Server

• Should there be a constant running RuneScape test server where ninja fixes, patch notes, and other bug fixes are pushed a week or a few days in advance of release?
• Larger updates (i.e. POF, EDs, Solak, Skills), would be pushed on a case by case basis.
• With larger updates it would be entirely possible to limit the pool of players available and potentially sign them to NDA.
• How do you get people to play betas?

• You can't work with the same characters because potential unbalanced updates would make that unfair. You can work with characters that you can give any level you want, but that stops being fun much very quickly, and also skews the range of levels that the content is tested for.

• Benefits?

• A stream of players testing new patches, getting a jump on the Monday bugs that appear. If a bug is small it could be hotfixed in time for launch Monday.

• Drawbacks?

• Requires a rethink of Jagex’s development process. By Friday afternoon the update of the week is being packaged for launch. This process would need to be amenable and happen earlier in the week.
• Any time the player base is exposed to an update prior to launch, even with an NDA, they gain an advantage.

In-House Player Testing

• August through October 2018 saw numerous waves of players make their way through Jagex HQ as player testers.
• Players could apply, had to make their own way to Cambridge if selected, and sign an NDA.
• In game player testing was done for the most recent boss, Solak, it’s not known what updates were tested recently.
• Are in house player tests Jagex’s answer to the lack of a RuneScape test server?

• What if a group of people get access to beta servers. They have to play new content for 1 hour a week at minimum, and maybe in return they get free membership while they do so. It will never be a perfect representation (you get people who care about the game a lot), but the question is whether you can get an accurate cross section of the playerbase at all.

• Theory: Jagex wants to do more direct testing but the game update pipeline doesn’t allow for a beta server to be run easily. With in-house testing the environment is controlled, NDAs are signed, specific aspects of the game are tested, with the option for guided feedback.

The Mining & Smithing Beta

• In early 2018 players sampled a prototype of the Mining & Smithing rework. This was used to test the flow of the reworked skills and ensure the new methods (as prototypes) met player demand.
• At RuneFest, attendees could play a recent build of the Mining & Smithing rework in the full game world.
• This November, all players will be able to test the rework in the actual game world rather than a test square as we did in early 2018.
• Tips for Beta Success:

• Play the beta as you would your actual in-game account.
• Report bugs through official channels
• Bring feedback forward in an orderly manner through official channels
• Realize the play style of the rework is locked in. Focus on bugs, value of items, user interface, and polish.
• Evangelize the good. Bring the bad to Jagex, in civility.

• Is the beta even needed?

Show Data

• Episode Title: Building a Better Beta
• Teaser: We run through Jagex’s past history of beta content, what the EoC beta meant for RuneScape, RuneScape Test Servers, in house player testing, and set up the Mining & Smithing Beta. And, does building a better beta start with better feedback?
• Recorded Date: October 28, 2018
• Hosts: Shane, Tanis, and Cireon
• Duration: 1:15:17
• Editing Notes: Cireon coughing fit