Saturday, May 17, 2008
ION08 Conference - Joe Ludwig on Pirates of the Burning Sea
Joe Ludwig gave a talk at ION 2008 called "Pirates of the Burning Sea: A Post Partum." These are the notes I took on the talk, any mistakes and misinterpretations are my own. My comments are in brackets.
Pirates of the Burning Sea (PotBS): Standard $50+$15/mo subscription model. 4 nations, 5 careers, 3 sword fighting schools, player driven economy, tactical ship combat as star feature.
Project started in October 2002, 4 people, 6-12 month schedule. Planning to do something pretty small, get it out there, small core user base, gradually grow it.
Launched Jan 22 2008, 70+ pepole, more than 5 years of development.
How did that happen?
Under a year left on project for the whole time they were developing. We'd make some progress, grow the scope, repeat. We'd fool ourselves into thinking it's 6 months instead of 12, then revise the estimate back to 12. We never decided to make a big game.
Lesson 1: know how big your game is when you start. Don't start small and grow the scope. Problem with scope creep: implement features that are scaled down to meet the unrealistic schedule, then years later you rip them out and put in the feature you should have had. Never the right time for long-term investment. Why make tools if shiipping "soon"? Never work on anything requiring more than a month of programmer time (although actually you spend 6 months extending it month to month). Also never a good time to hire a new programmer because it takes 3 months to do hiring, then another 3 to train up. It's also never a good time to fire anybody!
Upside of scope creep: if we started with a big project we never would have even tried in the first place. It's boiling frog development, based on the possibly apocryphal story that if you put a frog in boiling water it will jump out, but if you put it in warm water and heat it until it's boiling, it won't jump out to save itself.
Lesson 2: figure out what game you're making as quickly as possible. The relationship with publisher usually drives this, but PotBS didn't have a publisher really until the last 8 months. Self-funded, so no tight vision required.
In fact, there were four visions: lead designer, executive producer, content lead, lead programmer.
Lead designer: deep combat sim at the very core, realistic historic setting. [Designer was in the audience: "I thought it was cool at the time."] Cannon fired (range table for realistic ballistics) -> did I hit the ship? -> which hull section (12 sections)? -> did I hit crew? -> did I hit superstructure? -> if not, did it hit the crew on the other side of the ship? -> did it hit the armor on the other side of the ship? REPEAT THESE CALCS FOR SPLINTERS THAT COME OFF THE HULL!
Executive Producer: an MMO but with good graphics. Started in 2002, so it was going up against UO, EQ, DAoC. Didn't recognize at the time that good graphics meant big budget. Wanted naval traditions. And a Starfleet Command style combat system.
Content Lead: rich single-player content. Storytelling emphasis. Scenarios respond to player actions and tell interesting stories.
Lead Programmer [Joe Ludwig, the speaker]: 6 month project! Minimal feature set that will grow after launch. Keep costs down. Developed reputation as the "NO" guy. Not necessarily a good thing either, even after it was clear the game wasn't small, he still kept pushing for small scale solutions and made bad investments.
They lived with four visions for two years!
Meanwhile they worked on a pitch for a scifi license game, realized that the scifi pitch was a better game than the pirate one. When scifi game fell through, they ported the design back to the pirate game. They created the overview spec, a 50 page document (probably a little short for such a big overview).
Result of overview spec: historical setting is kept. Kept realism. Not super hardcore but moreso than other MMOs. Starfleet Command style combat made it through. No ability to tune combat with the very complex ballistics. Kept naval traditions. Kept single player style content. Ditched the simulation, not just in ship combat but also ecnonmy. Threw out the 6 months deadline, realized it was a big game all of a sudden. [Oddly, they kept pretty much everything from all the visions except the sim.]
Time for a development reboot. For the first time they could think long-term and could afford to have the game not playable while doing infrastructure work.
They had a todo list for the first time since the scope was there. Eliminated entire categories of things we weren't doing: player-to-player combat, player-driven economy. (although they made it into the final game)
Still didn't have a single vision owner. Switch to new combat made the design team weak within the company politically. Vision was incomplete because it was pretty high level and also pretty short. Also had no art vision whatsoever. Had hired an artist but not an art director. Still paying the price for this: good graphics was the goal, so system specs were higher than they should've been. Needlessly complex ship models. Naval tradition is good for history buffs but they had to deemphasize piracy to a certain extent. Single player content focus caused lack of focus on group play.
Lesson 3: don't make scheduling harder than it has to be.
Impossible to really schedule the game due to unrealistic scope and planning, but they tried some stuff.
Tracking flat task lists in Excel, and a summary, one milestone at a time. Worked well when there were 6-8 people on team, but didn't support dependencies at all. You just had to remember stuff. Didn't scale over 15 people.
Critical Chain, a Methodology with lots of books and conferences and classes. Tracking every dependency between all tasks anywhere in your system. 2-3 month milestones, and used it from 20-50 people. But it had high overhead. Had someone spending a week of every month on the schedule. Unpredictable results! The purpose of CC is that it tells you on any day who the important person to allocate resources to. But sometimes 8 or 10 people were most critical according to CC! On a 40 person team that's crazy. Also you could have two people scheduled for the same task and one would be critical and one wouldn't. Hard to understand. Kept using CC well past the point where it was clearly hurting us.
Task lists in email. It's an agile thing, one month sprints. Little-a agile. Responds well to change, is lightweight. Doesn't track dependencies. The big thing as a part of this switch though was that we shortened the length of sprints from 2-3 months to 1 month. It's way easier to keep track of 1 month sprints and change direction as needed. Decentralized the scheduling. 8-40 hour tasks. Requires QA time to happen after the sprint is over. 3 week chunk ofwork, one week integration, 2 weeks on test server. Works well with 60-70 people.
Lesson 4: don't save polish for the end.
Don't fool yourself that beta is polish period. Launched pirates with thousands of known bugs. Testers were very good at finding bugs, but the programmers had no time to fix bugs. In last 6 months of development, only critical bugs ever got fixed [wow, crazy].
Next time I'd like to give people time to polish as they're working on things. Allow people to have enough time to finish a task for real. Prior to this programmers would leave edge cases in a state where they wouldn't crash but there'd be annoying problems. Not super common, but with thousands of players uncommon bugs come up a lot. Mission designers had same problem, always more content to create.
Lesson 5: don't abandon quality for quantity of content.
At some point we decided we needed 1000 missions for launch. We did it, we launched with it. But we repeated lots of content and the missions were pretty cookie cutter. It's been criticized by players.
Lesson 6: deal with client performance from the start.
What we did was build for a min spec that was getting less and less cutting edge as the project grew older. So that was okay, but next time around I want to make the game run on all the laptops in this room, even the tiny little Thinkpads. You really have to define the art budgets to make the game look great on the min spec. Artists should test as they go and have the poly budgets set.
Lesson 7: make sure everyone is on the same side.
We had a problem during development where the mission designers and the testers had a bizarre war going on. Mission designers would make a mission but not polish it, give it to testers, they'd find the bugs, then keep entering trivial bugs. There was a message where the doctor asks you to get some rhubarb. As he said it "the local soil didn't work very well for rhubarb." Tester looks at port and says that the port has a resource "fertile soil", files a bug. It goes back to content, content bounces it back wontfix. Tester gets mad, flings it back, eventually management had to step in. Two departments developed an antagonistic relationship. On the QA side it was really just one problem tester.
Lesson 8: don't be afraid to fire people who aren't working out.
Fired 5 people in 5 years. People who fight against the team aren't worth keeping. Sometimes they make the people they work with uncomfortable and unproductive. Just not worth keeping.
Lesson 9: company culture is important.
People like working at Fying Labs. People get teased, lots of photoshopping, afternoon coffee train. We have an open door policy that's taken pretty seriously. Anyone can talk to anyone all the way up to the CEO about anything. How's the game doing, are we going to finish on time, what about benefits, etc. Another thing that's helped us out is we have a CEO who's genuinely enthusiastic about what we're working on. I mentioned that we were close to shipping a number of times, and at any one of those times at another company we might have started crunching. That's not something that we've done. We avoid it liuke the plague. Maybe a day or two at the end of a sprint, rarely the entire team, and rarely the same person from sprint to sprint. We never schedule people for 7 days a week or expect people to come out on the weekend. So we've had really low turnover. 5 firings, lost 3 more, out of 70 people.
Lesson 10: your closed beta should not be as long as ours.
Our closed beta was two years. Didn't get too much actionable feedback because the game was too much in flux, and burned out key community members. Distracted the team from implementing core systems to deal with feedback anyway even thought the feedback wasn't very useful. Also, the problem with the feedback was we never got a large enough beta population for most of the beta because we knew we were burning out the community: we didn't want to lose EVERYONE!
Lesson 11: hire some experienced people for key positions.
A little experience goes a long way. Almost no MMO launch experience. One animator, content director, community director had launch experience. No programmers, no designers, no operations [but Joe said after the talk that the launch was pretty good. they did have large site experience but not MMO experience], no QA. In a lot of cases didn't have game experience or even WORK experience. Art director is a teacher at the Art Institute, so art department staffed by the best student interns who convert to contractors to full time. great pipeline, but every single person through the pipeline was entry level. Median experience level of art department was very low. Content department almost the same way, but even the experienced folks were not experienced in content, had converted rom other departemnts.
They're experienced now!
Lesson 12: build more tools. [yeah, he didn't say any more than that!!]
Lesson 13: hire a great art director.
We hired ours halfway through, you should start with one.
Lesson 14: players love character customization.
We have 17 customization slots, a dozen pieces per slot, two colors per piece, millions of combinations, most of it available at character creation. Players spent almost an hour in character creation. Part of the reason that works is they divorced stats from appearance, so nobody has to look ugly to be more effective. On the other hand, it makes status stuff harder, you can't look at someone and gauge their level. But specific clothing options are unlocked by mission content to make up for that. Including peglegs!
lesson 15: unique systems are both good and bad
Ship combat is very well polished and tuned. It really works, is best system in the game. I'd hold it up to the best system in any game. People love it, they talk about it, they strategize. It places emphasis on player skill, which is big with PVPers. On the other hand it's nothing like any other MMO combat ever. Which makes it hard to teach. People don't understand it: it's a little too weird. It's slower placed than many games' combat systems. It also requires more attention than most mmo combat almost paradoxically: way more variables to keep track of. Especially at the high end game.
lesson 16: don't try to cram an entire new combat system in in a year.
Avatar combat not nearly as polished as ship combat. Content and system built in parallel and they don't work so well. Post-launch content for the system is great because you understand the system. No iteration time on the system, though.
lesson 17: love your community and they will love you back.
We've got a very active community relations program. Everybody writes devlogs, everybody posts to the forums. When we make a mistake, tune something poorly, we own up to it, and we tell people what we're doing to fix it. We listen to the players, we don't give them veto power over design decisions but we do listen to them, and we really made relations with the community everybody's job.
The result is that even negative reviews of the game would talk about how responsive Flying Labs is to the community. This also means that players who are often frustrated tend to give us the benefit of the doubt. They will often respond to something they don't like with "I don't like it because ____" as opposed to "you all suck, die in a fire". This makes our forums comparatively civil. We still have flame wars but they're better in general. People are nicer to each other and to us. Our community director is a big part of that.
[Audience Q&A started here]
Q: How did you make the funding without a publisher?
a: We funded ourselves by being lucky enough to be self-funded.
Q: What were your marketing challenges?
a: One of the big ones was the combat style. Some reviewers get it, some reviewers don't. Somewhat difficult learning curve at first. People just don't totally get the game.
q: Automated testing?
a: Early on we had a programmer whose job was to build an automated test system.
q: What's the frequency of post-launch content updates given the sprint system?
a: 4-5 weeks per update.
q: You mentioned that your design team was politically weakened after the combat system change. How was that eventually resolved?
a: It was resolved, don't want to get to specific. We made some personnel changes.
q: Do you feel the game was ready at launch?
a: It was in pretty good shape. Could always be better. A couple months past launch would have been great, but without launching it would not have become great. Needed the live feedback. would have liked to do a larger scale closed beta before launch.
Pirates of the Burning Sea (PotBS): Standard $50+$15/mo subscription model. 4 nations, 5 careers, 3 sword fighting schools, player driven economy, tactical ship combat as star feature.
Project started in October 2002, 4 people, 6-12 month schedule. Planning to do something pretty small, get it out there, small core user base, gradually grow it.
Launched Jan 22 2008, 70+ pepole, more than 5 years of development.
How did that happen?
Under a year left on project for the whole time they were developing. We'd make some progress, grow the scope, repeat. We'd fool ourselves into thinking it's 6 months instead of 12, then revise the estimate back to 12. We never decided to make a big game.
Lesson 1: know how big your game is when you start. Don't start small and grow the scope. Problem with scope creep: implement features that are scaled down to meet the unrealistic schedule, then years later you rip them out and put in the feature you should have had. Never the right time for long-term investment. Why make tools if shiipping "soon"? Never work on anything requiring more than a month of programmer time (although actually you spend 6 months extending it month to month). Also never a good time to hire a new programmer because it takes 3 months to do hiring, then another 3 to train up. It's also never a good time to fire anybody!
Upside of scope creep: if we started with a big project we never would have even tried in the first place. It's boiling frog development, based on the possibly apocryphal story that if you put a frog in boiling water it will jump out, but if you put it in warm water and heat it until it's boiling, it won't jump out to save itself.
Lesson 2: figure out what game you're making as quickly as possible. The relationship with publisher usually drives this, but PotBS didn't have a publisher really until the last 8 months. Self-funded, so no tight vision required.
In fact, there were four visions: lead designer, executive producer, content lead, lead programmer.
Lead designer: deep combat sim at the very core, realistic historic setting. [Designer was in the audience: "I thought it was cool at the time."] Cannon fired (range table for realistic ballistics) -> did I hit the ship? -> which hull section (12 sections)? -> did I hit crew? -> did I hit superstructure? -> if not, did it hit the crew on the other side of the ship? -> did it hit the armor on the other side of the ship? REPEAT THESE CALCS FOR SPLINTERS THAT COME OFF THE HULL!
Executive Producer: an MMO but with good graphics. Started in 2002, so it was going up against UO, EQ, DAoC. Didn't recognize at the time that good graphics meant big budget. Wanted naval traditions. And a Starfleet Command style combat system.
Content Lead: rich single-player content. Storytelling emphasis. Scenarios respond to player actions and tell interesting stories.
Lead Programmer [Joe Ludwig, the speaker]: 6 month project! Minimal feature set that will grow after launch. Keep costs down. Developed reputation as the "NO" guy. Not necessarily a good thing either, even after it was clear the game wasn't small, he still kept pushing for small scale solutions and made bad investments.
They lived with four visions for two years!
Meanwhile they worked on a pitch for a scifi license game, realized that the scifi pitch was a better game than the pirate one. When scifi game fell through, they ported the design back to the pirate game. They created the overview spec, a 50 page document (probably a little short for such a big overview).
Result of overview spec: historical setting is kept. Kept realism. Not super hardcore but moreso than other MMOs. Starfleet Command style combat made it through. No ability to tune combat with the very complex ballistics. Kept naval traditions. Kept single player style content. Ditched the simulation, not just in ship combat but also ecnonmy. Threw out the 6 months deadline, realized it was a big game all of a sudden. [Oddly, they kept pretty much everything from all the visions except the sim.]
Time for a development reboot. For the first time they could think long-term and could afford to have the game not playable while doing infrastructure work.
They had a todo list for the first time since the scope was there. Eliminated entire categories of things we weren't doing: player-to-player combat, player-driven economy. (although they made it into the final game)
Still didn't have a single vision owner. Switch to new combat made the design team weak within the company politically. Vision was incomplete because it was pretty high level and also pretty short. Also had no art vision whatsoever. Had hired an artist but not an art director. Still paying the price for this: good graphics was the goal, so system specs were higher than they should've been. Needlessly complex ship models. Naval tradition is good for history buffs but they had to deemphasize piracy to a certain extent. Single player content focus caused lack of focus on group play.
Lesson 3: don't make scheduling harder than it has to be.
Impossible to really schedule the game due to unrealistic scope and planning, but they tried some stuff.
Tracking flat task lists in Excel, and a summary, one milestone at a time. Worked well when there were 6-8 people on team, but didn't support dependencies at all. You just had to remember stuff. Didn't scale over 15 people.
Critical Chain, a Methodology with lots of books and conferences and classes. Tracking every dependency between all tasks anywhere in your system. 2-3 month milestones, and used it from 20-50 people. But it had high overhead. Had someone spending a week of every month on the schedule. Unpredictable results! The purpose of CC is that it tells you on any day who the important person to allocate resources to. But sometimes 8 or 10 people were most critical according to CC! On a 40 person team that's crazy. Also you could have two people scheduled for the same task and one would be critical and one wouldn't. Hard to understand. Kept using CC well past the point where it was clearly hurting us.
Task lists in email. It's an agile thing, one month sprints. Little-a agile. Responds well to change, is lightweight. Doesn't track dependencies. The big thing as a part of this switch though was that we shortened the length of sprints from 2-3 months to 1 month. It's way easier to keep track of 1 month sprints and change direction as needed. Decentralized the scheduling. 8-40 hour tasks. Requires QA time to happen after the sprint is over. 3 week chunk ofwork, one week integration, 2 weeks on test server. Works well with 60-70 people.
Lesson 4: don't save polish for the end.
Don't fool yourself that beta is polish period. Launched pirates with thousands of known bugs. Testers were very good at finding bugs, but the programmers had no time to fix bugs. In last 6 months of development, only critical bugs ever got fixed [wow, crazy].
Next time I'd like to give people time to polish as they're working on things. Allow people to have enough time to finish a task for real. Prior to this programmers would leave edge cases in a state where they wouldn't crash but there'd be annoying problems. Not super common, but with thousands of players uncommon bugs come up a lot. Mission designers had same problem, always more content to create.
Lesson 5: don't abandon quality for quantity of content.
At some point we decided we needed 1000 missions for launch. We did it, we launched with it. But we repeated lots of content and the missions were pretty cookie cutter. It's been criticized by players.
Lesson 6: deal with client performance from the start.
What we did was build for a min spec that was getting less and less cutting edge as the project grew older. So that was okay, but next time around I want to make the game run on all the laptops in this room, even the tiny little Thinkpads. You really have to define the art budgets to make the game look great on the min spec. Artists should test as they go and have the poly budgets set.
Lesson 7: make sure everyone is on the same side.
We had a problem during development where the mission designers and the testers had a bizarre war going on. Mission designers would make a mission but not polish it, give it to testers, they'd find the bugs, then keep entering trivial bugs. There was a message where the doctor asks you to get some rhubarb. As he said it "the local soil didn't work very well for rhubarb." Tester looks at port and says that the port has a resource "fertile soil", files a bug. It goes back to content, content bounces it back wontfix. Tester gets mad, flings it back, eventually management had to step in. Two departments developed an antagonistic relationship. On the QA side it was really just one problem tester.
Lesson 8: don't be afraid to fire people who aren't working out.
Fired 5 people in 5 years. People who fight against the team aren't worth keeping. Sometimes they make the people they work with uncomfortable and unproductive. Just not worth keeping.
Lesson 9: company culture is important.
People like working at Fying Labs. People get teased, lots of photoshopping, afternoon coffee train. We have an open door policy that's taken pretty seriously. Anyone can talk to anyone all the way up to the CEO about anything. How's the game doing, are we going to finish on time, what about benefits, etc. Another thing that's helped us out is we have a CEO who's genuinely enthusiastic about what we're working on. I mentioned that we were close to shipping a number of times, and at any one of those times at another company we might have started crunching. That's not something that we've done. We avoid it liuke the plague. Maybe a day or two at the end of a sprint, rarely the entire team, and rarely the same person from sprint to sprint. We never schedule people for 7 days a week or expect people to come out on the weekend. So we've had really low turnover. 5 firings, lost 3 more, out of 70 people.
Lesson 10: your closed beta should not be as long as ours.
Our closed beta was two years. Didn't get too much actionable feedback because the game was too much in flux, and burned out key community members. Distracted the team from implementing core systems to deal with feedback anyway even thought the feedback wasn't very useful. Also, the problem with the feedback was we never got a large enough beta population for most of the beta because we knew we were burning out the community: we didn't want to lose EVERYONE!
Lesson 11: hire some experienced people for key positions.
A little experience goes a long way. Almost no MMO launch experience. One animator, content director, community director had launch experience. No programmers, no designers, no operations [but Joe said after the talk that the launch was pretty good. they did have large site experience but not MMO experience], no QA. In a lot of cases didn't have game experience or even WORK experience. Art director is a teacher at the Art Institute, so art department staffed by the best student interns who convert to contractors to full time. great pipeline, but every single person through the pipeline was entry level. Median experience level of art department was very low. Content department almost the same way, but even the experienced folks were not experienced in content, had converted rom other departemnts.
They're experienced now!
Lesson 12: build more tools. [yeah, he didn't say any more than that!!]
Lesson 13: hire a great art director.
We hired ours halfway through, you should start with one.
Lesson 14: players love character customization.
We have 17 customization slots, a dozen pieces per slot, two colors per piece, millions of combinations, most of it available at character creation. Players spent almost an hour in character creation. Part of the reason that works is they divorced stats from appearance, so nobody has to look ugly to be more effective. On the other hand, it makes status stuff harder, you can't look at someone and gauge their level. But specific clothing options are unlocked by mission content to make up for that. Including peglegs!
lesson 15: unique systems are both good and bad
Ship combat is very well polished and tuned. It really works, is best system in the game. I'd hold it up to the best system in any game. People love it, they talk about it, they strategize. It places emphasis on player skill, which is big with PVPers. On the other hand it's nothing like any other MMO combat ever. Which makes it hard to teach. People don't understand it: it's a little too weird. It's slower placed than many games' combat systems. It also requires more attention than most mmo combat almost paradoxically: way more variables to keep track of. Especially at the high end game.
lesson 16: don't try to cram an entire new combat system in in a year.
Avatar combat not nearly as polished as ship combat. Content and system built in parallel and they don't work so well. Post-launch content for the system is great because you understand the system. No iteration time on the system, though.
lesson 17: love your community and they will love you back.
We've got a very active community relations program. Everybody writes devlogs, everybody posts to the forums. When we make a mistake, tune something poorly, we own up to it, and we tell people what we're doing to fix it. We listen to the players, we don't give them veto power over design decisions but we do listen to them, and we really made relations with the community everybody's job.
The result is that even negative reviews of the game would talk about how responsive Flying Labs is to the community. This also means that players who are often frustrated tend to give us the benefit of the doubt. They will often respond to something they don't like with "I don't like it because ____" as opposed to "you all suck, die in a fire". This makes our forums comparatively civil. We still have flame wars but they're better in general. People are nicer to each other and to us. Our community director is a big part of that.
[Audience Q&A started here]
Q: How did you make the funding without a publisher?
a: We funded ourselves by being lucky enough to be self-funded.
Q: What were your marketing challenges?
a: One of the big ones was the combat style. Some reviewers get it, some reviewers don't. Somewhat difficult learning curve at first. People just don't totally get the game.
q: Automated testing?
a: Early on we had a programmer whose job was to build an automated test system.
q: What's the frequency of post-launch content updates given the sprint system?
a: 4-5 weeks per update.
q: You mentioned that your design team was politically weakened after the combat system change. How was that eventually resolved?
a: It was resolved, don't want to get to specific. We made some personnel changes.
q: Do you feel the game was ready at launch?
a: It was in pretty good shape. Could always be better. A couple months past launch would have been great, but without launching it would not have become great. Needed the live feedback. would have liked to do a larger scale closed beta before launch.
Labels: ION 2008, postmortem
Everyday Shooter
Jon Mak's Everyday Shooter is the greatest game I have played in a very long time. It's too soon to tell, but it may end up in my short list of "top 5 favorite games of all time." I've been playing the game, well, every day since it was released on Steam about 9 days ago. And I've been itching to write an article explaining how important this game is to me for almost that entire time. This is my attempt at expressing that feeling.
Odd Reverberations
I get the music from ES stuck in my head all the time. This isn't that weird for a video game... except that ES's music is generative based on how you play the game. Have you ever had a nondeterministic piece of music stuck in your head? If you listen to modern composers like Steve Reich, you probably have. Let me tell you, it's a strange, synaesthetic experience.
So I'll have the general chord progression of a song like the opening level, "Robot," going through my head. And then I'll imagine the embellishments coming in: the individual guitar plucks and licks. But when I imagine the sound it's coupled to the image of the enemy ships exploding away in a sheen of light. My brain can't recall the sound without recalling the light. This is marvelous and exhilarating: no game has ever done this to me before.
Lush Look Killer
The third level of Everyday Shooter is called "Lush Look Killer." It's pretty damned hard, especially at first. Here's the level:
You'll notice that as you first enter the level, this creepy eye right in the center is closed, and then... robots start to exit the eye, and walk back into it, growing the eye. I played the first minute of the level probably a dozen times before understanding it enough to continue. Here's how it went, where each bullet point is something new I learned on each playthrough:
Oh, and of course the level design superbly follows the tone of the music. The verse is funky, yet almost plodding, suggesting the march of the robots and the fact that you're not really in any danger. Yet it's ominous: that eyeball is going to open up sometime, and distant distortion gets louder as you approach that moment. And when it does, you hit the chorus: roomy, powerful guitar strumming and utter chaos as you're inundated with creepy eyeballs. I could go on about the rest of the level but you get the idea, and of course you can watch the video to get an even better idea.
Incidentally, there's a nice progression of zone-of-safety in the first three levels: in "Robot," the edges are dangerous and the middle is safe. In "Root of the Heart" the edges are your safe zone, all the danger is in the middle. In "Lush Look Killer" the edges and the middle are dangerous: you have to basically circle strafe the eyeball while avoiding the edges. Come to think of it, "Porco in the Sky" ups the ante and gives you two targets to circle strafe right off the bat. It's a nice little progression and reinforces the idea of game-as-album.
Hmmm....
I've written a lot here, but I still don't know: why do I keep playing this game? I guess I'll think about it and post some more later.
Odd Reverberations
I get the music from ES stuck in my head all the time. This isn't that weird for a video game... except that ES's music is generative based on how you play the game. Have you ever had a nondeterministic piece of music stuck in your head? If you listen to modern composers like Steve Reich, you probably have. Let me tell you, it's a strange, synaesthetic experience.
So I'll have the general chord progression of a song like the opening level, "Robot," going through my head. And then I'll imagine the embellishments coming in: the individual guitar plucks and licks. But when I imagine the sound it's coupled to the image of the enemy ships exploding away in a sheen of light. My brain can't recall the sound without recalling the light. This is marvelous and exhilarating: no game has ever done this to me before.
Lush Look Killer
The third level of Everyday Shooter is called "Lush Look Killer." It's pretty damned hard, especially at first. Here's the level:
You'll notice that as you first enter the level, this creepy eye right in the center is closed, and then... robots start to exit the eye, and walk back into it, growing the eye. I played the first minute of the level probably a dozen times before understanding it enough to continue. Here's how it went, where each bullet point is something new I learned on each playthrough:
- OMG WTF eyeballs and robots?
- Okay, if I kill the flashing robot it acts like a smartbomb, killing all the other robots.
- Huh, the robots grow the eyeball as they return back to it.
- Oh crap, the more the eyeball grows the more robots it sends out! I'd better take advantage of those big flashing robots to keep from being overrun with more.
- Wait, the robots aren't really attacking me. Can I just wait over here and let the eyeball grow? Okay, the eyeball's gigantic. It's opening up... crap now there's a million tiny eyeballs! (Dies)
- Alright, this time I'll keep the eyeball small. Hey look, this time there are fewer tiny eyeballs. I guess the bigger the main eyeball, the more small eyeballs spawn.
- Wait, some of these eyeballs are being transformed by the big eyeball into homing eyeballs! And there are these other eyeballs with force fields around them?
- Ah, I can slowly grow the force field eyes so their field gets bigger and bigger by shooting them, but not to fast or they'll pop. But when they do pop, they take out all enemies in their field!
- Hey, not only do they take out enemies, but they send out homing missiles and it appears that there are more missiles the bigger I grow the field before popping it.
- ...hey, I can make the big eyeball smaller by shooting it directly.
Oh, and of course the level design superbly follows the tone of the music. The verse is funky, yet almost plodding, suggesting the march of the robots and the fact that you're not really in any danger. Yet it's ominous: that eyeball is going to open up sometime, and distant distortion gets louder as you approach that moment. And when it does, you hit the chorus: roomy, powerful guitar strumming and utter chaos as you're inundated with creepy eyeballs. I could go on about the rest of the level but you get the idea, and of course you can watch the video to get an even better idea.
Incidentally, there's a nice progression of zone-of-safety in the first three levels: in "Robot," the edges are dangerous and the middle is safe. In "Root of the Heart" the edges are your safe zone, all the danger is in the middle. In "Lush Look Killer" the edges and the middle are dangerous: you have to basically circle strafe the eyeball while avoiding the edges. Come to think of it, "Porco in the Sky" ups the ante and gives you two targets to circle strafe right off the bat. It's a nice little progression and reinforces the idea of game-as-album.
Hmmm....
I've written a lot here, but I still don't know: why do I keep playing this game? I guess I'll think about it and post some more later.
Labels: everyday shooter
Thursday, May 15, 2008
ION08 Conference - Scaling on a Dime Panel
These are my notes from the Scaling On A Dime panel at ION 2008. The participants (along with their letter abbreviations) were as follows:
Marty Poulin (M) (moderator)
Joe Ludwig (J)
Victor Jimenez (V)
Brian Hafer (B)
Larry Mellon (L)
Any mistakes or misinterpretations are my own damn fault. Don't blame these guys. My comments in square brackets.
M: Intros?
L: Been in distributed computing for over 20 years, research programmer for 10 years in early days of microcomputer and ethernet, found taht dev challenges were hard, so focused on tools, supercomputing, virtual world researcher for DARPA, then joined EA and worked on The Sims Online by bringing military production expertise in. Then cofounded Emergent Game Technologies.
J: Been at Flying Labs for 8.5 years, all the way through to production of Pirates of the Burning Sea.
B: Got into online long before games, worked at startups before working at Emergent on metrics product, currently at Tubemogul dealing with rapid scaling issues for online service in web 2.0 space.
V: Work for Northrop Grummond. Does distributed tech for simulations, not interested in the dime since it's govt, but TIME is a big factor in implementation.
M: The reason we're talking about scaling on a dime instead of random testing stuff is that there's a new project lifecycle: prototype, beta, iterate til traction, scale or fail! Minimize time and costs, ensure scale, and have good quality of service. Have to scale people as well as infrastructure: devs, tools, pipeline, process. Licensing vs development: open source (OS) and middleware pros and cons. OS/MW can be faster/cheaper, might be better than what you can build, reduced maintenance, but also could give vendor dependence.
What are the processes you use to pick middleware or OS?
L: Sometimes different pieces of middleware are built with different assumptions in mind, so integration errors become a very big problem as we use more and more pieces of software. do your scaling during testing as opposed to scaling live with users. For OS, find groups with momentum and not much politics. You want to know where your middleware is going to be not just now but in 5 years when you launch.
J: We had a similar problem. Our graphics engine provider went bankrupt 6 months after we licensed them. We thought we were close to launch, so we stuck with them, game turned out to take 4.5 more years all the while with no vendor in existence! You need to do due diligence with vendors, and also do code escrow so you can get source code if they go out of business.
B: In the OS space, there are new projects every day, you can always find what seems to be a perfect fit niche project, but they won't necessarily be mature and robust. Look at the reliability record. Public release or beta software? I'd recommend not adopting anything except the more well-backed, tried-and-true projects.
L: I got burned once by not investigating the testing practices of the OS project, turned out that the engineer just ran it a few times on his desktop to test.
V: We find we have very long term projects, 15-20 years [wow!]. When we evaluate the project, we have to look at ease of use of package, documentation, etc. Not so obvious is that you have to look at the metaphor behind it: if the metaphor doesn't fit you end up writing a lot of glue code. Find a project that is going in the same direction as you. You may go through lots of evals but in the long run it really pays off. Make sure it's multithreaded before you license it! Other minor ones: make sure the source you're getting from OS is actually readable so you can debug.
J: Unreadable source is not unique to OS. We licensed a commercial graphics engine that had copy and paste code with one line in each copy that was 1600 characters long!
B: Even best case if the source is good, having to track down bugs is still difficult and a sinkhole of time and money trying to track down bugs because you may not understand the architecture. Source code is not a silver bullet.
J: But you don't usually get source code to change the middleware. You get it to debug or trace: when you get a middleware error it's usually your fault but you still have to be able to trace the code to learn what you did wrong.
L: This is why you need test rigs for the entire system as well as individual modules so you can very easily track down the source of the problem.
M: It's not only about whether the code is good, but what about support? Any nightmares?
L: At EA we bought a very expensive build distribution package. 8 builds being sent around to about 120 devs, 200 QA, and a bunch of executives. We were pushing so much data around at such high rates, they'd never anticipated this in the distribution package so it couldn't handle it. We had to reengineer their system on the fly!
J: Our support has been pretty good, but before we started on PotBS and were evaluating different engines. One middleware provider asked to visit to evaluate the engine. We said, "Okay, let's go this month." They replied "we're too busy this month shipping Unreal Tournament. You have to visit next month." Big red flag that they won't support us in a time of need. [Epic has since hired a whole department to support middleware so in theory this doesn't happen anymore, but see the Silicon Knights lawsuit for claims to the contrary.]
M: What about legal issues with licensing?
V: I work for the government. Part of problem is that the gov't noticed that games are an interesting technology. Serious games, RFIs, RFPs, etc that deal with game technology. That's a new infusion of money and talent, great. However, gov't has very interesting ideas about what intellectual property (IP) means. one of the things that happens to people that has caught many unawares is that if the gov't gives you money and you develop something with their money it is now the government's IP to do as they will. So you make your MMO, spend $3M of your money on it, gov't gives you $200k to put in one feature, you deliver, and the gov't takes the whole thing and distributes it to anybody they want. The gov't says, "Hey, your IP is infected with your rights now. Nyah nyah." Catches large companies as well as small.
L: Licensing issue in OS is slowing down its adoption. I've had to turn down promising packages with how the license infects the rest of our middleware. I've heard of game teams that had to pull out OS from games because lawyers are afraid of launchtime security exploits due to problems with the OS paackage having exposed source [which is kind of weird, but whatev].
M: What do you like in OS?
L: Simple things. GCC. Anything that's a tool is very safe to use. Lawyers concerned with software to release to customers, but in-house is good. [what about in-house infrastructure that is part of the production environment, such as any messaging systems etc?]
J: Not so much OS, but we like PathEngine, Rad Game Tools on the middleware side.
M: How do we actually get things to scale? Selecting components is fine but it doesn't mean they're going to scale to the levels we're going to push them to. What role does testing play in the evaluation of this software?
L: I've grown to become a big proponent of testing. Started with automated testing in the 80s on supercomputing. Talked about test rigs for components, but you also need automation of testing of the system as a whole. The middleware network transport we used had an individual test that showed it worked under 15k clients under normal loads, but integrated with the whole system only supported 200 clients!!
J: Testing has worked well for us. We have a test dev who does nothing but test tools and automation. We learned from Larry's GDC talk [in 2003]. We have lots of test servers identical to production servers but with client bots that hit them with load.
V: From day one we were an extreme programming shop. Always follow test first philosophy. Every single line of code has a passfail unit test, run through the test cases. We have a suite of tests which involves people manually running through stuff in specialized virtual cockpits, and above that we actual operational equipment to test on. By the time it leaves the lab it's well tested. Too many times we've been bitten by the bug that when you get OS or middleware that is buggy and won't be fixed for a while by provider. Don't trust the provider.
L: Test suites are great but they're still code, so there can be bugs in your test suites. Calibrate the accuracy of the test system and the game itself. Measure level of nondeterminism in your test system gives you an idea of the accuracy of your tests!! [dude hells yeah]
M: What are some rules to fix the scaling problems?
J: Avoid single points of failure, obviously. We try to make it so we can bring additional resources to bear on any of the problems we face. Front line that talks to the client are certain servers that they can scale independently and also increases reliability because they're independent processes.
B: Build a service oriented architecture rather than monolithic tightly coupled system. Be sure to define your interface points and where your components will talk to each other so you can scale over processing power and then problem become data access shared across tons of users (high availability).
L: Metrics can help find where that bottleneck is and who's using the data, how often, what frequency, to see if it's a time-dependent bottleneck (which is harder) or not. You can architect around these bottlenecks if you understand them.
J: We have things on PotBS that are a little more monolithic than they should be. Limits our ability to scale. We have to have more clusters than we might otherwise need because some systems need to be just so. Very hard to do this stuff after you're up and running.
M: Coupling is a lesson learned by experienced engineers.
B: Verify your engineers actually follow your architecture plans!!
L: We forced our engineers to avoid coupling. They broke the architecture because they went through DLL boundaries.
J: Have a test mode where nothing is ever on the same server so that every call that could be a remote process is forced to go to be a remote process, that way you have to architect it right.
M: Up to this point when were putting together server architectures for MMOs, in order to figure out our infrastructure we have to take and educated guess. Look at competition and market size, interest that's out there, and we make a guess. We hope that when people arrive that we're right. If we're wrong, we either spent too much money and go out of business, or we didn't scale enough and our services are not working and we're out of business.
Animoto, a video service, was fine with 50 server instances and then overnight they shot up to 3000 server instances! Any rules of thumb to figure out what you're shooting for?
L: Run extended load test derived from actual beta test numbers. But you can't really guess how many users you're going to get.
J: We can take a pretty good guess as to how many users a cluster can support but we can't predict the number of users.
M: We've all had experience with the epic fail. From my standpoint, when we released we didn't even have the right tested bandwidth for WW2 Online.
L: We had the opposite failure. We hit 100k players in 2 or 3 months but we'd projected 1M users and bought hardware for 1M users in 4 months.
V: I know ahead of time how many users I will have. I know where they're located, etc. But I never really know what they're going to do. They're virtual cockpits for new airplanes and we literally don't know what they're going to do. Govt wants it to work perfectly but I don't know what the bandwidth or processing requirements are, so I have to guess from the plane's features what the requirements will be.
M: With no good user profiling you can't guess that.
L: you can actually change the game to tune the user behavior to lower server requirements.
J: Make the game less fun to perform better? A an alternate example is EQ1 got a feature where you could keep your character online as a vendor, but that meant there was WAY MORE CONCURRENCY but didn't result in more play time.
M: New thing on horizon is the idea of scaling via cloud computing. Google, MS, Joyent, Amazon Web Services EC2 is premier right now. But lots of tradeoffs. Great because you can scale by throwing money at the problem, initially more services for less money, easy to set up. But you have less contrtol, sometimes QOS issues, higher costs at scale. What makes using cloud computing system suitable for a particular business?
J: One of the thing that people talk about is you depend on your provider to be up all the time. For certain businesses--Amazon just had many hours of downtime on S3 [yikes!], so people are concerned about uptime. Fact is that the uptime of amazon or google will completely blow away uptime of random game studio x, so for small studio not an issue, but for EA it might matter since EA can have very good uptime.
B: Size and resources drive the decision. If you are small or moderate sized, you can get great bang for buck and better reliability. But if you're huge like EA, amazon is charging a markup, you'd do better with your own economy of scale and get more control over your operations. We use amazon at my current company to support scaling. It's very reliable, but there is no legal QOS guarantee. Have been instances such as with S3 that took even amazon by surprise. They were unprepared to deal with it. They were not prepared to communicate what was going on. But you can't beat it on the cost model.
L: Unless something goes wrong. Some guy got hacked, his BW bill went up--
B: This was a friend of mine who built a service had an exploit so someone was using his instances to run rogue processes, got saddled with $60k bill from Amazon at the end of month!
L: When using these things, put your own triggers and alarms on service usage.
M: Ensure your monetization model supports scale. more users may just put you out of business.
J: On the other hand, with Amazon in particular, you or a piece of code you control has to ask for extra resources. If you're willing to let QOS suffer or have users in queues to keep costs down, you can do that if you want.
M: Typically with mmos, rather than have bad QOS we put people in the queue. It says to customers not "we are bad" but "we are not available".
J: If there are flash crowds or certain server resources that being taxed, we will queue people to get around that.
L: If you are trying to scale on a dime, maybe avoid hard problems like a seamless virtual world from the very start.
M: each service has a different api, a different model, so to a certain extent there's vender lockin.
L: We use presentation layers where I expose what I need to have and I can write glue code, and evaluate new vendors.
B: With Google you're tied into their API, but with Amazon it's really more of a virtual machine you're running. VM options are very flexible.
J: But there are tradeoffs. If you're using VMs with Amazon, you have to have an operations staff and you own the installs for the operating systems. With Ggoogle appengine it's higher layer. You're writing python and it's running on their infrastructure that you don't care about. You have to deploy builds but not operate hundreds of VMs.
M: This brings about complexity issue. If we have 3000 VMs, is it any easier to go with a service than managing real or virtual servers yourself?
J: It's a lot easier to go with Amazon because you can get a new VM set up in 5 minutes on a web form, as opposed to physically installing a new machine at the data center and all the associated overhead.
B: Services like Amazon force you to have horizontally scalable architecture right from the beginning. If you're not architected right, more VMs won't help you at all.
[Audience questions follow]
Q: Regarding scalability and quality. have any of you built a solution in the cloud that overcomes issues where servers die and you spin up a new one without users noticing?
B: We use amazon EC2, and you have to deal with instance failure. Minimize what you're storing on an instance. Use instances for computing power rather than data storage. S3 is permanent long-term, They're launching something that's more like a local hard disk in the future. If you're storing data on your disk, if you're running a DB server on EC2, you have serious problems.
Q: What if one part of system is self hosted, one is in cloud? What about latency?
B: We're not seeing a lot of latency problems between us and the cloud. We run data locally and processing in the cloud. You can put data in the cloud but you have to architect more robustly.
Q: You said infrastructure design is hard to do up front. What about heuristics on the best things to do to prepare for scaling? How do you plan for user peak?
J: More of a marketing question that they can't answer very well. Preorder programs can't really help.
M: In the end, nobody knows.
J: For traditional mmos, you look at preorder and 2.6x that for initial launch week population. But your preorder program can be too high or too low for a wide variety of reasons. We had difficulty getting preorder boxes on shelves, so our preorder was delayed by a month so we couldn't really tell. If you have to go one way or other: underbuy resources. Lines out the door are better than a big empty building.
M: One last point: as you scale with Amazon, they're competitive at lower scale, but as you scale up Amazon gets more expensive and you'll be better off doing it yoursel.
J: Brian, do you have cost numbers for EC2?
B: Check out their website, it's up there.
Marty Poulin (M) (moderator)
Joe Ludwig (J)
Victor Jimenez (V)
Brian Hafer (B)
Larry Mellon (L)
Any mistakes or misinterpretations are my own damn fault. Don't blame these guys. My comments in square brackets.
M: Intros?
L: Been in distributed computing for over 20 years, research programmer for 10 years in early days of microcomputer and ethernet, found taht dev challenges were hard, so focused on tools, supercomputing, virtual world researcher for DARPA, then joined EA and worked on The Sims Online by bringing military production expertise in. Then cofounded Emergent Game Technologies.
J: Been at Flying Labs for 8.5 years, all the way through to production of Pirates of the Burning Sea.
B: Got into online long before games, worked at startups before working at Emergent on metrics product, currently at Tubemogul dealing with rapid scaling issues for online service in web 2.0 space.
V: Work for Northrop Grummond. Does distributed tech for simulations, not interested in the dime since it's govt, but TIME is a big factor in implementation.
M: The reason we're talking about scaling on a dime instead of random testing stuff is that there's a new project lifecycle: prototype, beta, iterate til traction, scale or fail! Minimize time and costs, ensure scale, and have good quality of service. Have to scale people as well as infrastructure: devs, tools, pipeline, process. Licensing vs development: open source (OS) and middleware pros and cons. OS/MW can be faster/cheaper, might be better than what you can build, reduced maintenance, but also could give vendor dependence.
What are the processes you use to pick middleware or OS?
L: Sometimes different pieces of middleware are built with different assumptions in mind, so integration errors become a very big problem as we use more and more pieces of software. do your scaling during testing as opposed to scaling live with users. For OS, find groups with momentum and not much politics. You want to know where your middleware is going to be not just now but in 5 years when you launch.
J: We had a similar problem. Our graphics engine provider went bankrupt 6 months after we licensed them. We thought we were close to launch, so we stuck with them, game turned out to take 4.5 more years all the while with no vendor in existence! You need to do due diligence with vendors, and also do code escrow so you can get source code if they go out of business.
B: In the OS space, there are new projects every day, you can always find what seems to be a perfect fit niche project, but they won't necessarily be mature and robust. Look at the reliability record. Public release or beta software? I'd recommend not adopting anything except the more well-backed, tried-and-true projects.
L: I got burned once by not investigating the testing practices of the OS project, turned out that the engineer just ran it a few times on his desktop to test.
V: We find we have very long term projects, 15-20 years [wow!]. When we evaluate the project, we have to look at ease of use of package, documentation, etc. Not so obvious is that you have to look at the metaphor behind it: if the metaphor doesn't fit you end up writing a lot of glue code. Find a project that is going in the same direction as you. You may go through lots of evals but in the long run it really pays off. Make sure it's multithreaded before you license it! Other minor ones: make sure the source you're getting from OS is actually readable so you can debug.
J: Unreadable source is not unique to OS. We licensed a commercial graphics engine that had copy and paste code with one line in each copy that was 1600 characters long!
B: Even best case if the source is good, having to track down bugs is still difficult and a sinkhole of time and money trying to track down bugs because you may not understand the architecture. Source code is not a silver bullet.
J: But you don't usually get source code to change the middleware. You get it to debug or trace: when you get a middleware error it's usually your fault but you still have to be able to trace the code to learn what you did wrong.
L: This is why you need test rigs for the entire system as well as individual modules so you can very easily track down the source of the problem.
M: It's not only about whether the code is good, but what about support? Any nightmares?
L: At EA we bought a very expensive build distribution package. 8 builds being sent around to about 120 devs, 200 QA, and a bunch of executives. We were pushing so much data around at such high rates, they'd never anticipated this in the distribution package so it couldn't handle it. We had to reengineer their system on the fly!
J: Our support has been pretty good, but before we started on PotBS and were evaluating different engines. One middleware provider asked to visit to evaluate the engine. We said, "Okay, let's go this month." They replied "we're too busy this month shipping Unreal Tournament. You have to visit next month." Big red flag that they won't support us in a time of need. [Epic has since hired a whole department to support middleware so in theory this doesn't happen anymore, but see the Silicon Knights lawsuit for claims to the contrary.]
M: What about legal issues with licensing?
V: I work for the government. Part of problem is that the gov't noticed that games are an interesting technology. Serious games, RFIs, RFPs, etc that deal with game technology. That's a new infusion of money and talent, great. However, gov't has very interesting ideas about what intellectual property (IP) means. one of the things that happens to people that has caught many unawares is that if the gov't gives you money and you develop something with their money it is now the government's IP to do as they will. So you make your MMO, spend $3M of your money on it, gov't gives you $200k to put in one feature, you deliver, and the gov't takes the whole thing and distributes it to anybody they want. The gov't says, "Hey, your IP is infected with your rights now. Nyah nyah." Catches large companies as well as small.
L: Licensing issue in OS is slowing down its adoption. I've had to turn down promising packages with how the license infects the rest of our middleware. I've heard of game teams that had to pull out OS from games because lawyers are afraid of launchtime security exploits due to problems with the OS paackage having exposed source [which is kind of weird, but whatev].
M: What do you like in OS?
L: Simple things. GCC. Anything that's a tool is very safe to use. Lawyers concerned with software to release to customers, but in-house is good. [what about in-house infrastructure that is part of the production environment, such as any messaging systems etc?]
J: Not so much OS, but we like PathEngine, Rad Game Tools on the middleware side.
M: How do we actually get things to scale? Selecting components is fine but it doesn't mean they're going to scale to the levels we're going to push them to. What role does testing play in the evaluation of this software?
L: I've grown to become a big proponent of testing. Started with automated testing in the 80s on supercomputing. Talked about test rigs for components, but you also need automation of testing of the system as a whole. The middleware network transport we used had an individual test that showed it worked under 15k clients under normal loads, but integrated with the whole system only supported 200 clients!!
J: Testing has worked well for us. We have a test dev who does nothing but test tools and automation. We learned from Larry's GDC talk [in 2003]. We have lots of test servers identical to production servers but with client bots that hit them with load.
V: From day one we were an extreme programming shop. Always follow test first philosophy. Every single line of code has a passfail unit test, run through the test cases. We have a suite of tests which involves people manually running through stuff in specialized virtual cockpits, and above that we actual operational equipment to test on. By the time it leaves the lab it's well tested. Too many times we've been bitten by the bug that when you get OS or middleware that is buggy and won't be fixed for a while by provider. Don't trust the provider.
L: Test suites are great but they're still code, so there can be bugs in your test suites. Calibrate the accuracy of the test system and the game itself. Measure level of nondeterminism in your test system gives you an idea of the accuracy of your tests!! [dude hells yeah]
M: What are some rules to fix the scaling problems?
J: Avoid single points of failure, obviously. We try to make it so we can bring additional resources to bear on any of the problems we face. Front line that talks to the client are certain servers that they can scale independently and also increases reliability because they're independent processes.
B: Build a service oriented architecture rather than monolithic tightly coupled system. Be sure to define your interface points and where your components will talk to each other so you can scale over processing power and then problem become data access shared across tons of users (high availability).
L: Metrics can help find where that bottleneck is and who's using the data, how often, what frequency, to see if it's a time-dependent bottleneck (which is harder) or not. You can architect around these bottlenecks if you understand them.
J: We have things on PotBS that are a little more monolithic than they should be. Limits our ability to scale. We have to have more clusters than we might otherwise need because some systems need to be just so. Very hard to do this stuff after you're up and running.
M: Coupling is a lesson learned by experienced engineers.
B: Verify your engineers actually follow your architecture plans!!
L: We forced our engineers to avoid coupling. They broke the architecture because they went through DLL boundaries.
J: Have a test mode where nothing is ever on the same server so that every call that could be a remote process is forced to go to be a remote process, that way you have to architect it right.
M: Up to this point when were putting together server architectures for MMOs, in order to figure out our infrastructure we have to take and educated guess. Look at competition and market size, interest that's out there, and we make a guess. We hope that when people arrive that we're right. If we're wrong, we either spent too much money and go out of business, or we didn't scale enough and our services are not working and we're out of business.
Animoto, a video service, was fine with 50 server instances and then overnight they shot up to 3000 server instances! Any rules of thumb to figure out what you're shooting for?
L: Run extended load test derived from actual beta test numbers. But you can't really guess how many users you're going to get.
J: We can take a pretty good guess as to how many users a cluster can support but we can't predict the number of users.
M: We've all had experience with the epic fail. From my standpoint, when we released we didn't even have the right tested bandwidth for WW2 Online.
L: We had the opposite failure. We hit 100k players in 2 or 3 months but we'd projected 1M users and bought hardware for 1M users in 4 months.
V: I know ahead of time how many users I will have. I know where they're located, etc. But I never really know what they're going to do. They're virtual cockpits for new airplanes and we literally don't know what they're going to do. Govt wants it to work perfectly but I don't know what the bandwidth or processing requirements are, so I have to guess from the plane's features what the requirements will be.
M: With no good user profiling you can't guess that.
L: you can actually change the game to tune the user behavior to lower server requirements.
J: Make the game less fun to perform better? A an alternate example is EQ1 got a feature where you could keep your character online as a vendor, but that meant there was WAY MORE CONCURRENCY but didn't result in more play time.
M: New thing on horizon is the idea of scaling via cloud computing. Google, MS, Joyent, Amazon Web Services EC2 is premier right now. But lots of tradeoffs. Great because you can scale by throwing money at the problem, initially more services for less money, easy to set up. But you have less contrtol, sometimes QOS issues, higher costs at scale. What makes using cloud computing system suitable for a particular business?
J: One of the thing that people talk about is you depend on your provider to be up all the time. For certain businesses--Amazon just had many hours of downtime on S3 [yikes!], so people are concerned about uptime. Fact is that the uptime of amazon or google will completely blow away uptime of random game studio x, so for small studio not an issue, but for EA it might matter since EA can have very good uptime.
B: Size and resources drive the decision. If you are small or moderate sized, you can get great bang for buck and better reliability. But if you're huge like EA, amazon is charging a markup, you'd do better with your own economy of scale and get more control over your operations. We use amazon at my current company to support scaling. It's very reliable, but there is no legal QOS guarantee. Have been instances such as with S3 that took even amazon by surprise. They were unprepared to deal with it. They were not prepared to communicate what was going on. But you can't beat it on the cost model.
L: Unless something goes wrong. Some guy got hacked, his BW bill went up--
B: This was a friend of mine who built a service had an exploit so someone was using his instances to run rogue processes, got saddled with $60k bill from Amazon at the end of month!
L: When using these things, put your own triggers and alarms on service usage.
M: Ensure your monetization model supports scale. more users may just put you out of business.
J: On the other hand, with Amazon in particular, you or a piece of code you control has to ask for extra resources. If you're willing to let QOS suffer or have users in queues to keep costs down, you can do that if you want.
M: Typically with mmos, rather than have bad QOS we put people in the queue. It says to customers not "we are bad" but "we are not available".
J: If there are flash crowds or certain server resources that being taxed, we will queue people to get around that.
L: If you are trying to scale on a dime, maybe avoid hard problems like a seamless virtual world from the very start.
M: each service has a different api, a different model, so to a certain extent there's vender lockin.
L: We use presentation layers where I expose what I need to have and I can write glue code, and evaluate new vendors.
B: With Google you're tied into their API, but with Amazon it's really more of a virtual machine you're running. VM options are very flexible.
J: But there are tradeoffs. If you're using VMs with Amazon, you have to have an operations staff and you own the installs for the operating systems. With Ggoogle appengine it's higher layer. You're writing python and it's running on their infrastructure that you don't care about. You have to deploy builds but not operate hundreds of VMs.
M: This brings about complexity issue. If we have 3000 VMs, is it any easier to go with a service than managing real or virtual servers yourself?
J: It's a lot easier to go with Amazon because you can get a new VM set up in 5 minutes on a web form, as opposed to physically installing a new machine at the data center and all the associated overhead.
B: Services like Amazon force you to have horizontally scalable architecture right from the beginning. If you're not architected right, more VMs won't help you at all.
[Audience questions follow]
Q: Regarding scalability and quality. have any of you built a solution in the cloud that overcomes issues where servers die and you spin up a new one without users noticing?
B: We use amazon EC2, and you have to deal with instance failure. Minimize what you're storing on an instance. Use instances for computing power rather than data storage. S3 is permanent long-term, They're launching something that's more like a local hard disk in the future. If you're storing data on your disk, if you're running a DB server on EC2, you have serious problems.
Q: What if one part of system is self hosted, one is in cloud? What about latency?
B: We're not seeing a lot of latency problems between us and the cloud. We run data locally and processing in the cloud. You can put data in the cloud but you have to architect more robustly.
Q: You said infrastructure design is hard to do up front. What about heuristics on the best things to do to prepare for scaling? How do you plan for user peak?
J: More of a marketing question that they can't answer very well. Preorder programs can't really help.
M: In the end, nobody knows.
J: For traditional mmos, you look at preorder and 2.6x that for initial launch week population. But your preorder program can be too high or too low for a wide variety of reasons. We had difficulty getting preorder boxes on shelves, so our preorder was delayed by a month so we couldn't really tell. If you have to go one way or other: underbuy resources. Lines out the door are better than a big empty building.
M: One last point: as you scale with Amazon, they're competitive at lower scale, but as you scale up Amazon gets more expensive and you'll be better off doing it yoursel.
J: Brian, do you have cost numbers for EC2?
B: Check out their website, it's up there.
Labels: conferences, infrastructure, ION 2008
ION08 Conference - Brandon Reinhart on Conveying Your Vision
Brandon Reinhart of SpaceTime Studios gave a talk called Using Storytelling to Craft and Convey Vision. These are my session notes, mistakes and misinterpretations are my own so blame me for those. My comments in brackets.
How many designers in the audience have worked on an asset brought through the pipeline that in no way turned out the way you wanted? This talk will help you avoid that through storytelling and visualization techniques. Brandon's a lead designer at SpaceTime in Austin working on a future fantasy MMO ground/space hybrid, $15m investment, moving into production now. Brandon's responsible for IP development, content design, systems design, wrote the story with a partner, day to day design production.
How to be a visionary designer. Most of our work takes place in our heads away from our computers, in shower, playing games, driving, etc. We build content in our minds and we can even play our game experience in our heads. Traditionally that process of getting the design communicated happens through the design doc. We delegate systems and say "document it." What we get is "a big book of stupid." Nobody reads the design doc except designers! Poor method of conveying ideas. The reason visionary designers are visionary is that they realize that design docs mean nothing if nobody understands what you're talking about.
Not on a crusade to destroy the design doc. Publishers want them. They're good at creating asset lists, scoping, and speccing game systems. But design docs won't get a game made. We have to develop more fundamental descriptors of our design ideas.
Most commonly you write a linear story about game experience to convey your idea, but still not very effective. Two core principles of visionary designer:
One: thou shalt tailor thine delivery to thine audience. Should know your audience. We often go through a creative design process, but through that we obfuscate the core concepts that drive our game. We should convey the core concepts and educate the audience with that FIRST.
Two: thou shalt engender buy-in. [So that the audience perceives the vision as their own.] When you're talking to someone about a cool idea, anything from an asset all the way to a game, if you pound them over the head they won't be interested. If you can engender buy-in, a sense of participation, they'll go so far as to do your work FOR you. Not just "here's our idea what do you think?" but break it down and then get feedback on that.
Why narrative design? Illusion of transparency is the enemy. When we look at our writing we think intent is clear, but it's not so much to our audience. We need make sure they understand fundamental concepts: describing emotions you want to convey is better than speccing out a whole document. A picture can be worth a thousand words.
Stories create mental images, mental images create real images, real images communicate concepts. Designers and concept artists should team up. Normally concept art is separated from designers by a few producers and managers. At SpaceTime there's a a direct connection from designer to concept artist: CRP is a Concept Request Package. Contains a narrative description (one paragraph to two pages), scope of story scales with scope of asset. CRP links to material in wiki format. Eventually can back off on narrative because concept artists will come to "get it".
Sometimes you have to make 150 blasters, and concepters make 150 random blasters. But designers should care about controlling some of the direction and detail. Don't write 150 stories about blasters, but write maybe 15 stories about cultural background of the different blaster companies in your game. [I love this idea, it's similar to building a robust fictional world in general]
Tool 1: the concept pyramid
When pitching something large scale, we have a tendency to allow our passion for something to completely dominate our delivery. We'll go off on weird tangents with pointless detail. Tailor your delivery to fundamental concepts. Concept pyramid is an exercise in your mind to get back to those fundaments. Futuristic scifi game: top of pyramid is SPACE WAR! Second level down: humanity, demonic aliens. Third level: humanity is about idealism and destiny, aliens are about religious prophesy, and both are about tech vs magic.
[random idea: "any sufficiently crappy magic is indistinguishable from technology", apologies to the late Arthur C Clarke]
As your audience gets more familiar you break down level by level as they ask more questions and GET things.
Tool 2: key moments
A picture is worth a thousand words. A key moment is a one-image storyboard. Translate your mental images into the visual language of the game. Fuels the viewer's imagination during a pitch.
Let's say we're DC Online and trying to convey things to someone. Good position: all comic covers are key moments, so we'd make a scene that has Batman swinging through a window with thugs shooting and a damsel in distress with the Joker snarling. Has suspense, unknown outcome, aspiration ("I want to be Batman! Or the Joker!")
Artists should look into speed paintings. 20 minute sketch, discuss, paint over, discuss, iterate on that and create a very detailed on-point image in about 4 hours.
In fantasy games, people understand the tropes, you don't have to be a visionary communicator. players just unerstand what a fighter is. But these key moment sketches can help during character gen for a less traditional game (for example) to help people understand what the expectations are for the characters that they're going to be playing.
tool 3: aspiration driven character design
Fantasy games get this for free. Characters speak to a player's aspirations. You have to convey the character's values without relying on words.
Start with aspiration: wealth, power, virility. One single word. Then write a few sentences about the aspiration. What does this character want and fear? Write this in the universe of your world. Tie it in to gameplay systems a little bit too.
Don't be a douche when you're conveying your vision! It's fundamentally a social and political activity. You want to be knowledegable and talented, but also humble. Distill down concepts, incorporate feedback, avoid defensiveness, revise and rewrite. Always be willing to say, "wait, this is crap."
[This is all very similar to the original BioShock pitch that Ken Levine discussed at the Boston Post Mortem in August '07: they built a few rooms of interactive super polished concept art and that was the pitch.]
How many designers in the audience have worked on an asset brought through the pipeline that in no way turned out the way you wanted? This talk will help you avoid that through storytelling and visualization techniques. Brandon's a lead designer at SpaceTime in Austin working on a future fantasy MMO ground/space hybrid, $15m investment, moving into production now. Brandon's responsible for IP development, content design, systems design, wrote the story with a partner, day to day design production.
How to be a visionary designer. Most of our work takes place in our heads away from our computers, in shower, playing games, driving, etc. We build content in our minds and we can even play our game experience in our heads. Traditionally that process of getting the design communicated happens through the design doc. We delegate systems and say "document it." What we get is "a big book of stupid." Nobody reads the design doc except designers! Poor method of conveying ideas. The reason visionary designers are visionary is that they realize that design docs mean nothing if nobody understands what you're talking about.
Not on a crusade to destroy the design doc. Publishers want them. They're good at creating asset lists, scoping, and speccing game systems. But design docs won't get a game made. We have to develop more fundamental descriptors of our design ideas.
Most commonly you write a linear story about game experience to convey your idea, but still not very effective. Two core principles of visionary designer:
One: thou shalt tailor thine delivery to thine audience. Should know your audience. We often go through a creative design process, but through that we obfuscate the core concepts that drive our game. We should convey the core concepts and educate the audience with that FIRST.
Two: thou shalt engender buy-in. [So that the audience perceives the vision as their own.] When you're talking to someone about a cool idea, anything from an asset all the way to a game, if you pound them over the head they won't be interested. If you can engender buy-in, a sense of participation, they'll go so far as to do your work FOR you. Not just "here's our idea what do you think?" but break it down and then get feedback on that.
Why narrative design? Illusion of transparency is the enemy. When we look at our writing we think intent is clear, but it's not so much to our audience. We need make sure they understand fundamental concepts: describing emotions you want to convey is better than speccing out a whole document. A picture can be worth a thousand words.
Stories create mental images, mental images create real images, real images communicate concepts. Designers and concept artists should team up. Normally concept art is separated from designers by a few producers and managers. At SpaceTime there's a a direct connection from designer to concept artist: CRP is a Concept Request Package. Contains a narrative description (one paragraph to two pages), scope of story scales with scope of asset. CRP links to material in wiki format. Eventually can back off on narrative because concept artists will come to "get it".
Sometimes you have to make 150 blasters, and concepters make 150 random blasters. But designers should care about controlling some of the direction and detail. Don't write 150 stories about blasters, but write maybe 15 stories about cultural background of the different blaster companies in your game. [I love this idea, it's similar to building a robust fictional world in general]
Tool 1: the concept pyramid
When pitching something large scale, we have a tendency to allow our passion for something to completely dominate our delivery. We'll go off on weird tangents with pointless detail. Tailor your delivery to fundamental concepts. Concept pyramid is an exercise in your mind to get back to those fundaments. Futuristic scifi game: top of pyramid is SPACE WAR! Second level down: humanity, demonic aliens. Third level: humanity is about idealism and destiny, aliens are about religious prophesy, and both are about tech vs magic.
[random idea: "any sufficiently crappy magic is indistinguishable from technology", apologies to the late Arthur C Clarke]
As your audience gets more familiar you break down level by level as they ask more questions and GET things.
Tool 2: key moments
A picture is worth a thousand words. A key moment is a one-image storyboard. Translate your mental images into the visual language of the game. Fuels the viewer's imagination during a pitch.
Let's say we're DC Online and trying to convey things to someone. Good position: all comic covers are key moments, so we'd make a scene that has Batman swinging through a window with thugs shooting and a damsel in distress with the Joker snarling. Has suspense, unknown outcome, aspiration ("I want to be Batman! Or the Joker!")
Artists should look into speed paintings. 20 minute sketch, discuss, paint over, discuss, iterate on that and create a very detailed on-point image in about 4 hours.
In fantasy games, people understand the tropes, you don't have to be a visionary communicator. players just unerstand what a fighter is. But these key moment sketches can help during character gen for a less traditional game (for example) to help people understand what the expectations are for the characters that they're going to be playing.
tool 3: aspiration driven character design
Fantasy games get this for free. Characters speak to a player's aspirations. You have to convey the character's values without relying on words.
Start with aspiration: wealth, power, virility. One single word. Then write a few sentences about the aspiration. What does this character want and fear? Write this in the universe of your world. Tie it in to gameplay systems a little bit too.
Don't be a douche when you're conveying your vision! It's fundamentally a social and political activity. You want to be knowledegable and talented, but also humble. Distill down concepts, incorporate feedback, avoid defensiveness, revise and rewrite. Always be willing to say, "wait, this is crap."
[This is all very similar to the original BioShock pitch that Ken Levine discussed at the Boston Post Mortem in August '07: they built a few rooms of interactive super polished concept art and that was the pitch.]
Labels: conferences, design, ION 2008, production
Wednesday, May 14, 2008
ION08 Conference - Dana Hanna on Shadowrun
Dana Hanna gave a 30-minute post mortem of Shadowrun titled You Can't Do That!
These are my notes, mistakes and misinterpretations are my own. I showed up a little late so I missed the first 5 minutes. My comments in brackets.
Shadowrun
One of the problems with developing the game was that multiple communities needed to be brought together: FPS, original RPG fans, etc.
The big reveal was at E3 2006. The game required some in-depth explanation, they couldn't just put it up and say "here you go, play it" [seems like a core design problem to me]
Audience: traditional game press, console gamers, pc gamers, probably not so much the tabletop gamers even though it should have been. The reactions:
Why did they go to E3 at all if the game wasn't polished? They believed in the game. [That's not enough!] To be completely honest: Hanna says it was the best game she ever worked on in terms of her own enjoyment. 1-2 hours a day of playtesting in the studio during development. The 60-100 person team would fight every day to get to be in the playtest. [This is normally a VERY good sign, see the AutoAssault post mortem from last year's OGDC for more on this]. "Our thoughts were probably colored a little by our enthusiasm for the game." Some team members were very big into the tabletop RPG. They lost sight of big picture in E3 crunch, community was not even on the radar.
Started the game with a small core team 18 months before E3, custom engine, crossplatform. Then, 8 months before E3 they did a HARD RESET on the whole game, largely to do with the art direction. Went multiplayer-only in this stage, lost most of the art team, new art team largely from film industry which was a tough transition. The first level and characters were finished DAYS before E3.
Post-E3: had to reassure the hardcore RPGers. Get people to talk about the game we were making instead of the game we were not making. Sometimes you work on games you don't like, but Hanna really liked this game.
Had to stop being a console game with a website and start being a real community. Had to fess up to some bad calls: went to E3 without talking about story, without engaging in the community about storylines and canon they felt attached to. Had to share their excitement.
Step one: invited over some important community folks.
Microsoft folks came first. An MS employee posted publicly that they were ashamed to be associated with MS due to Shadowrun E3 demo. The team sent out a meeting invite to him and said, "why don't you spend a day playing the game with us?" 10 seconds later accepted and asked to bring 2 friends. They were test cases. They set aside one day for online journalists, one day for community folks. Invited friendlies, neutrals, and naysayers. Visitors had lots of one-on-one with designers. hands-on training with team members [hmmm, hands-on training can be misleading], played the game a lot.
Established personal relationships with opinion leaders this way. got timely positive coverage and gained some credibility from people talking about gameplay itself as opposed to differences with RPG.
"I hate you. Can I get a beta invite?" Still a lot of open hostility after these community days, but everyone wanted in the beta, and they did let people in. Console betas don't happen often enough in general. Originally their beta focused entirely on technical networking issues. Grew into a PR event. Was a closed beta.
Beta takeaways: not nearly as gameplay focused as they should have been; instead they were technically focused. After a while the beta became a demo whether they liked it or not. Should have paid more attention to "hell is other people" [Sartre ftw]-- the game relies entirely on playing with your friends. The beta folks tended to know each other. Game wasn't nearly as fun in environment where people played with strangers. Beta was too late to put better matchmaking in.
Data collection only half the battle. Can data mine to your heart's content, have to understand what people are doing. Had TONS of data: longest sniper shots, how many people used given weapons etc. Couldn't say that people used rocket launcher to kill own teammates at spawn point [why not?]. Relied on community leaders for that kind of info. "If I could change something this time around I would like to find a more effective way to track what players are doing in game without relying so much on community leaders." [aha, okay, they just needed a better metrics system]
What did we learn? Did some things right. Brought community leaders into the family. Listened to feedback from RPG folks. After e3 took ideas from community for better continuity with canon. Welcomed well-behaved naysayers. What we'd do differently: should have engaged community earlier. Devote more resources to story. Run a PC beta. Add community-building features like clans or guilds.
In closing, Shadowrun was one of the most crunch-intensive games she'd ever worked on, 80-100 hr weeks for months on end, one of tightest teams ever worked with. Former devs still miss the FASA studio. Game has its flaws, still feels like best game she's ever worked on.
These are my notes, mistakes and misinterpretations are my own. I showed up a little late so I missed the first 5 minutes. My comments in brackets.
Shadowrun
One of the problems with developing the game was that multiple communities needed to be brought together: FPS, original RPG fans, etc.
The big reveal was at E3 2006. The game required some in-depth explanation, they couldn't just put it up and say "here you go, play it" [seems like a core design problem to me]
Audience: traditional game press, console gamers, pc gamers, probably not so much the tabletop gamers even though it should have been. The reactions:
- "why did you kill my baby?"
- FPS blasphemy
- set in Brazil? you work in Seattle! [Seattle is the main Shadowrun setting]
Why did they go to E3 at all if the game wasn't polished? They believed in the game. [That's not enough!] To be completely honest: Hanna says it was the best game she ever worked on in terms of her own enjoyment. 1-2 hours a day of playtesting in the studio during development. The 60-100 person team would fight every day to get to be in the playtest. [This is normally a VERY good sign, see the AutoAssault post mortem from last year's OGDC for more on this]. "Our thoughts were probably colored a little by our enthusiasm for the game." Some team members were very big into the tabletop RPG. They lost sight of big picture in E3 crunch, community was not even on the radar.
Started the game with a small core team 18 months before E3, custom engine, crossplatform. Then, 8 months before E3 they did a HARD RESET on the whole game, largely to do with the art direction. Went multiplayer-only in this stage, lost most of the art team, new art team largely from film industry which was a tough transition. The first level and characters were finished DAYS before E3.
Post-E3: had to reassure the hardcore RPGers. Get people to talk about the game we were making instead of the game we were not making. Sometimes you work on games you don't like, but Hanna really liked this game.
Had to stop being a console game with a website and start being a real community. Had to fess up to some bad calls: went to E3 without talking about story, without engaging in the community about storylines and canon they felt attached to. Had to share their excitement.
Step one: invited over some important community folks.
Microsoft folks came first. An MS employee posted publicly that they were ashamed to be associated with MS due to Shadowrun E3 demo. The team sent out a meeting invite to him and said, "why don't you spend a day playing the game with us?" 10 seconds later accepted and asked to bring 2 friends. They were test cases. They set aside one day for online journalists, one day for community folks. Invited friendlies, neutrals, and naysayers. Visitors had lots of one-on-one with designers. hands-on training with team members [hmmm, hands-on training can be misleading], played the game a lot.
Established personal relationships with opinion leaders this way. got timely positive coverage and gained some credibility from people talking about gameplay itself as opposed to differences with RPG.
"I hate you. Can I get a beta invite?" Still a lot of open hostility after these community days, but everyone wanted in the beta, and they did let people in. Console betas don't happen often enough in general. Originally their beta focused entirely on technical networking issues. Grew into a PR event. Was a closed beta.
Beta takeaways: not nearly as gameplay focused as they should have been; instead they were technically focused. After a while the beta became a demo whether they liked it or not. Should have paid more attention to "hell is other people" [Sartre ftw]-- the game relies entirely on playing with your friends. The beta folks tended to know each other. Game wasn't nearly as fun in environment where people played with strangers. Beta was too late to put better matchmaking in.
Data collection only half the battle. Can data mine to your heart's content, have to understand what people are doing. Had TONS of data: longest sniper shots, how many people used given weapons etc. Couldn't say that people used rocket launcher to kill own teammates at spawn point [why not?]. Relied on community leaders for that kind of info. "If I could change something this time around I would like to find a more effective way to track what players are doing in game without relying so much on community leaders." [aha, okay, they just needed a better metrics system]
What did we learn? Did some things right. Brought community leaders into the family. Listened to feedback from RPG folks. After e3 took ideas from community for better continuity with canon. Welcomed well-behaved naysayers. What we'd do differently: should have engaged community earlier. Devote more resources to story. Run a PC beta. Add community-building features like clans or guilds.
In closing, Shadowrun was one of the most crunch-intensive games she'd ever worked on, 80-100 hr weeks for months on end, one of tightest teams ever worked with. Former devs still miss the FASA studio. Game has its flaws, still feels like best game she's ever worked on.
Labels: conferences, design, ION 2008, metrics, postmortem
ION Conference Notes Aggregate Feed
So talking with Adam Martin here at the ION Conference, I realized that if we're both blogging our notes from sessions I might as well aggregate them together. So, using Yahoo! Pipes, I built this RSS feed of ION conference notes. If you're blogging notes from ION, let me know, make sure to use a keyphrase (like ION08 in the title of the blog post), and I'll add it to the feed!
Labels: conferences, ION 2008
ION08 Conference - Todd Northcutt on Buddy Lists
Todd Northcutt gave a talk at ION yesterday called Seven Cool Things You Can Do With Buddy Lists. Here are my notes from the session; he said that his slides will be posted at poweredbygamespy.com. My comments are in square brackets.
You can increase consumer game play and loyalty to a franchise by keeping gamers engaged while not actively playing.
Fun fact: Call of Duty 4 has no buddy list system for the PC!
Buddy lists are a means, not an end, and a buddy list system is not interesting by itself. Northcutt uses the phrases "connected gaming" to describe games that share stats, replays, and accomplishments but are not necessarily multiplayer or massively multiplayer.
Buddy lists create stickiness. Buddies can make you play games you may not even particularly love [is this a good thing?].
Three things that help stickiness: presence (see what your friends are doing), messaging, and easy invitations to online matches.
Contextual competition: leaderboards are irrelevant by themselves. On a typical leaderboard you are the 4,230th best player out of 846,111 players. If you take a leaderboard and filter it by just your friends on your buddy list, you're now the 2nd best player out of your 12 friends. That's way more powerful.
The non-core users care about stats too! You just have to make them relevant. For example, the default view on Mario Kart Wii is your friend score distribution, and even global stats highlight your friends.
You can use buddy lists to build a better matchmaker. If you can't play with friends you should at least play with people who play LIKE your friends. Perhaps give preference to FoaFs (friend-of-a-friend).
Break down barriers to continue the game experience outside the game itself. Crossmessaging, between platforms (different OSes, console/PC, etc). Let out of game people see that friends are playing to entice them in.
Broadcast messaging enables 1:many communication with all your buddies. If Joe joins game X, ping all his friends that "Joe just joined game x". Or send brags to your friends. Even game devs can communicate as buddies as another way to get messages out to players.
Enable commerce: let players know "your buddy has this cool thing: do you want it too? Click here to buy now!" You can sell full games, expansion packs, user trading, and microcontent this way. [Seems like I'd burn out on all the advertising, though.] Contextual "Buy
Now!" is super powerful.
Create a deeper and more rich social network. Help the game be more about relationships than content. Search for ways to create new relationships between people. Populate buddy lists better, import mail, [do local graph search for common FoaFs]. Provide easy introductions between users that are similar to encourage mingling with people you might not know but might like.
You can increase consumer game play and loyalty to a franchise by keeping gamers engaged while not actively playing.
Fun fact: Call of Duty 4 has no buddy list system for the PC!
Buddy lists are a means, not an end, and a buddy list system is not interesting by itself. Northcutt uses the phrases "connected gaming" to describe games that share stats, replays, and accomplishments but are not necessarily multiplayer or massively multiplayer.
Buddy lists create stickiness. Buddies can make you play games you may not even particularly love [is this a good thing?].
Three things that help stickiness: presence (see what your friends are doing), messaging, and easy invitations to online matches.
Contextual competition: leaderboards are irrelevant by themselves. On a typical leaderboard you are the 4,230th best player out of 846,111 players. If you take a leaderboard and filter it by just your friends on your buddy list, you're now the 2nd best player out of your 12 friends. That's way more powerful.
The non-core users care about stats too! You just have to make them relevant. For example, the default view on Mario Kart Wii is your friend score distribution, and even global stats highlight your friends.
You can use buddy lists to build a better matchmaker. If you can't play with friends you should at least play with people who play LIKE your friends. Perhaps give preference to FoaFs (friend-of-a-friend).
Break down barriers to continue the game experience outside the game itself. Crossmessaging, between platforms (different OSes, console/PC, etc). Let out of game people see that friends are playing to entice them in.
Broadcast messaging enables 1:many communication with all your buddies. If Joe joins game X, ping all his friends that "Joe just joined game x". Or send brags to your friends. Even game devs can communicate as buddies as another way to get messages out to players.
Enable commerce: let players know "your buddy has this cool thing: do you want it too? Click here to buy now!" You can sell full games, expansion packs, user trading, and microcontent this way. [Seems like I'd burn out on all the advertising, though.] Contextual "Buy
Now!" is super powerful.
Create a deeper and more rich social network. Help the game be more about relationships than content. Search for ways to create new relationships between people. Populate buddy lists better, import mail, [do local graph search for common FoaFs]. Provide easy introductions between users that are similar to encourage mingling with people you might not know but might like.
Labels: conferences, ION 2008
Monday, May 12, 2008
Two Panels @ ION Conference, May 13-14
So I'll be in Seattle this week, Monday night to Thursday night, attending the ION Conference, which was called OGDC last year.
I'm going to be part of two panel discussions. The first is on Wednesday at 9AM, Changing a Live Game: Lessons Learned and Techniques Applied. I'll be on the panel with Jason and Steve from 38 Studios, as well as Scott Hartsman who I met for the first time at IMGDC in March. Also on the panel is Osma Ahvenlampi, who I've never met before but is the CTO at Sulake (the Habbo Hotel guys). I'll be talking about using metrics to inform your changes to a live game.
The second panel is about metrics, called Tuning the Money Funnel: Customer and Process Metrics in Online Games. Larry Mellon is moderating--if you didn't know, he gave a seminal presentation on metrics for MMOs back at GDC 2003. He was pretty much the only name in metrics when I started doing that stuff. Also on the panel is Marty Poulin who I've known from GDC for a long time. This'll be my first time doing something "official" with him. Brian Hafer is rounding it out, and I've never met him so I'm looking forward to meeting him as well.
If you're in Seattle and want to meet up, drop me a line and I'll see what I can do!
I'm going to be part of two panel discussions. The first is on Wednesday at 9AM, Changing a Live Game: Lessons Learned and Techniques Applied. I'll be on the panel with Jason and Steve from 38 Studios, as well as Scott Hartsman who I met for the first time at IMGDC in March. Also on the panel is Osma Ahvenlampi, who I've never met before but is the CTO at Sulake (the Habbo Hotel guys). I'll be talking about using metrics to inform your changes to a live game.
The second panel is about metrics, called Tuning the Money Funnel: Customer and Process Metrics in Online Games. Larry Mellon is moderating--if you didn't know, he gave a seminal presentation on metrics for MMOs back at GDC 2003. He was pretty much the only name in metrics when I started doing that stuff. Also on the panel is Marty Poulin who I've known from GDC for a long time. This'll be my first time doing something "official" with him. Brian Hafer is rounding it out, and I've never met him so I'm looking forward to meeting him as well.
If you're in Seattle and want to meet up, drop me a line and I'll see what I can do!
Labels: conferences, speaking
Sunday, May 11, 2008
Paper Published on Spore's Animation System
Chris Hecker just posted an 11-page paper about the procedural animation system for Spore, authored by himself and other members of the Spore team. I'm downloading it now and will be giving it a read on the plane to the ION conference in Seattle tomorrow. Looking forward to reading it!
Labels: spore
Friday, May 09, 2008
Musical Montages in Video Games
So today at lunch I was discussing 80's movies with Jeff. In particular, I watched Real Genius last night and noted that there are 2 (maybe 2.5) musical training montages in that movie. That led to a pretty cool concept for advancing in a game.
So let's say your game gets to a point where your character has to become extremely badass in order to advance through the game. In RPGs, this sometimes means grinding for hours to become badass. In action games it may be an uber powerup that you get for a level. But what if you went through an 80s-movie musical training montage? It would play like WarioWare: tons of minigames, sometimes repeated, getting progressively harder, all to a tune like, Chaz Jankel's "Number One", or perhaps The C.S. Angels' "I'm Falling"? Make it all the way through the song a BAM you're a super-whatever-it-is.
This is mostly tongue-in-cheek but would be pretty awesome in a game that doesn't take itself too seriously. I could even see it working in an RPG, where there's a totally optional one-time item you can use where you summon a montage and gain three levels when you're done. Great for skipping the grind when you've lost your patience!
Also of note, Craig Perko first introduced me to the idea of musical montages in games as part of a LARP he wrote a few years ago called METEOR! Every character had one musical montage token to spend to do... basically whatever they wanted once during the game. As long as they role-played it!
And finally, I discovered this website, Movie-Montage.com, which is basically a database of musical montages. Yep. Wow.
So let's say your game gets to a point where your character has to become extremely badass in order to advance through the game. In RPGs, this sometimes means grinding for hours to become badass. In action games it may be an uber powerup that you get for a level. But what if you went through an 80s-movie musical training montage? It would play like WarioWare: tons of minigames, sometimes repeated, getting progressively harder, all to a tune like, Chaz Jankel's "Number One", or perhaps The C.S. Angels' "I'm Falling"? Make it all the way through the song a BAM you're a super-whatever-it-is.
This is mostly tongue-in-cheek but would be pretty awesome in a game that doesn't take itself too seriously. I could even see it working in an RPG, where there's a totally optional one-time item you can use where you summon a montage and gain three levels when you're done. Great for skipping the grind when you've lost your patience!
Also of note, Craig Perko first introduced me to the idea of musical montages in games as part of a LARP he wrote a few years ago called METEOR! Every character had one musical montage token to spend to do... basically whatever they wanted once during the game. As long as they role-played it!
And finally, I discovered this website, Movie-Montage.com, which is basically a database of musical montages. Yep. Wow.
Thursday, May 08, 2008
BEST NEWS OF YEAR
Everyday Shooter, a game I have rhapsodized about before, is now available on Steam after being PS3-exclusive for a year! $9. Buy it now. (via Jon Blow)
Labels: games
Monday, May 05, 2008
Macro-Indie Publishers
What are the implications of this new mega-indie-music publisher for the video game industry?
The short of the news is that 12,000 indie record labels have joined under one banner ("Merlin") to form a challenge to the Big Four publishers. One of the big reasons for this is so that indie labels can much more easily negotiate with iTunes and the other digital distribution networks.
Could this happen for indie games? I'm guessing not. This is an instance of a bunch of mini-publishers joining forces, and importantly they're not being acquired by a business entity. As far as I know, it's more of a trade group, so like the ESA represents lots of big game publishers, Merlin represents lots of little music publishers.
Indie game developers don't have publishers or labels. If they're the kind of indies who sell games on their own website, then they're analogous to musicians who do the same, or sell CDs at local shows. If an indie dev goes through Steam or Xbox Live or something... this is more like a single band negotiating with iTunes, or personally pitching Wal-Mart to carry their CD. Which just doesn't really happen.
I'm interested to learn more about Merlin and how they interact with their members, and what sort of control and power they have and choose to exert. How will they keep 12,000 member organizations happy? That's what I want to know.
The short of the news is that 12,000 indie record labels have joined under one banner ("Merlin") to form a challenge to the Big Four publishers. One of the big reasons for this is so that indie labels can much more easily negotiate with iTunes and the other digital distribution networks.
Could this happen for indie games? I'm guessing not. This is an instance of a bunch of mini-publishers joining forces, and importantly they're not being acquired by a business entity. As far as I know, it's more of a trade group, so like the ESA represents lots of big game publishers, Merlin represents lots of little music publishers.
Indie game developers don't have publishers or labels. If they're the kind of indies who sell games on their own website, then they're analogous to musicians who do the same, or sell CDs at local shows. If an indie dev goes through Steam or Xbox Live or something... this is more like a single band negotiating with iTunes, or personally pitching Wal-Mart to carry their CD. Which just doesn't really happen.
I'm interested to learn more about Merlin and how they interact with their members, and what sort of control and power they have and choose to exert. How will they keep 12,000 member organizations happy? That's what I want to know.
Wednesday, April 30, 2008
GTA
So tonight I go over to Darren's house to play GTA IV for the first time. As such, I was thinking a little bit about my past experiences with the GTA series, and one thing stood out.
I used to hate motorcyclists. Not personally, but when I was in my car and I'd see someone on a motorcycle, I would grumble. "Grr, what the hell, fucking biker," etc etc. Even if they weren't doing anything wrong, they'd just piss me off for no discernible reason.
Then I played GTA: San Andreas. I never played Vice City so it was my introduction to motorcycles in the GTA series. And let me tell you: it was a revelation. Motorcycles became my favorite way to get around, period. I must have played 60% of that game riding a motorcycle through all the different locales.
And around the time I was playing this, my attitude toward motorcyclists changed. Now when I see folks on motorcycles, I thinking, "Oh wow, that's probably a ton of fun." I have no desire to ride one myself, but I feel like I now understand where they're coming from.
So there you go. Grand Theft Auto: promoting peace, love, and understanding.
I used to hate motorcyclists. Not personally, but when I was in my car and I'd see someone on a motorcycle, I would grumble. "Grr, what the hell, fucking biker," etc etc. Even if they weren't doing anything wrong, they'd just piss me off for no discernible reason.
Then I played GTA: San Andreas. I never played Vice City so it was my introduction to motorcycles in the GTA series. And let me tell you: it was a revelation. Motorcycles became my favorite way to get around, period. I must have played 60% of that game riding a motorcycle through all the different locales.
And around the time I was playing this, my attitude toward motorcyclists changed. Now when I see folks on motorcycles, I thinking, "Oh wow, that's probably a ton of fun." I have no desire to ride one myself, but I feel like I now understand where they're coming from.
So there you go. Grand Theft Auto: promoting peace, love, and understanding.
Monday, April 28, 2008
How Do I Play Game?
Just caught this linked off of Gamasutra: how do i play game? is a blog of a non-gamer's experience playing the original Half-Life (well, probably the Source engine update). This guy has pretty much no experience gaming, and the blog reads like a really great usability playtest.
Definitely worth reading, especially if you wonder why people just don't "get" video games. Even the true classics have plenty of problems that keep them from ever becoming mass market phenomena.
Definitely worth reading, especially if you wonder why people just don't "get" video games. Even the true classics have plenty of problems that keep them from ever becoming mass market phenomena.
Labels: usability
More Creature Creator
So I just had lunch with Darren where we talked a little bit more about the Spore thing, and he brought up two points:
- They could certainly just include a $10 off coupon for the full game with a purchase of the full Creature Creator. That would defuse the complaints.
- And I hadn't mentioned this earlier because I thought it was obvious, but just in case: releasing the Creature Creator early is their method for filling the Spore pollinator servers with user-generated content, so that when people buy the full game in September there are already many hundreds of thousands of creatures.
Labels: business
Creature Creator Market Segmentation
There's been some Interwebs grumbling over the fact that EA will be offering two versions of the Spore Creature Creator demo. There will be a free version which offers 25% of all the creature parts, and there will be a $10 version with everything in the full game.
People are crying: "Oh no! This is a blatant attempt to suck every last penny out of us." Actually, I know some people who are not gamers and are not interested in the game aspects of Spore at all. The only thing they want to do is make creatures. What EA is doing is selling them the creature making aspect of the game, at the same price as a casual downloadable. There's still a free demo, folks.
I think the only major problem here is that of timing. If EA had released the free demo on June 17, and then released the $10 Creature Creator simultaneously with the release of the full game, nobody would be complaining.
Hmm. I started this blog post defending EA, and now I'm just unimpressed with their decision-making on this particular issue.
People are crying: "Oh no! This is a blatant attempt to suck every last penny out of us." Actually, I know some people who are not gamers and are not interested in the game aspects of Spore at all. The only thing they want to do is make creatures. What EA is doing is selling them the creature making aspect of the game, at the same price as a casual downloadable. There's still a free demo, folks.
I think the only major problem here is that of timing. If EA had released the free demo on June 17, and then released the $10 Creature Creator simultaneously with the release of the full game, nobody would be complaining.
Hmm. I started this blog post defending EA, and now I'm just unimpressed with their decision-making on this particular issue.
Labels: business
Tuesday, April 22, 2008
Funspot, Xybots
So this past weekend I went to Funspot with Darren. You might recognize Funspot as the huge classic games arcade where much of King of Kong takes place. Basically it's a giant arcade in lake region of New Hampshire where, for about $20 worth of tokens, you can play games (mostly from, say, '78 to '88) all day long.
In particular, I enjoyed playing Xybots, a game from 1987 that almost certainly was a direct influence on the early FPS games I loved so much like Wolfenstein 3D. I think what I like most about it is:
I'll definitely be going back to Funspot again. Hopefully I can bring a few people along with me!
In particular, I enjoyed playing Xybots, a game from 1987 that almost certainly was a direct influence on the early FPS games I loved so much like Wolfenstein 3D. I think what I like most about it is:
- Core simplicity. The core gameplay mechanics are dead simple: you have a gun, and you have a zapper (smart bomb). That's about it.
- Emergent complexity. There are many interesting situations you can get in, especially when you consider the incidental powerups (move faster, shoot faster, keys to open secret doors) that can be bought with a currency system.
- Cooperative play. I like playing this co-op with Darren (or with Jeff as I did at VXPO in 2006).
I'll definitely be going back to Funspot again. Hopefully I can bring a few people along with me!
Labels: games
The Final Days of Taldren
Erin Hoffman has a fantastic piece up on The Escapist about the final days of Taldren and their masterpiece-that-never-saw-the-light-of-day, Black9. It's a tale of a publisher (in this case, Majesco) royally screwing over a developer, using illegal bullying tactics, stealing source code, and of course, the batshit insane publisher-side producer.
You should go read it, because this is the kind of story you normally only hear about when you're 7 drinks in at midnight on Thursday at the Game Developers Conference.
As an aside, one of the books I recommend people read is Erik Bethke's Game Development and Production
. Bethke was one of the founders of Taldren, and his book actually includes some information on Black9 and lots of great information on how development process worked at Taldren.
You should go read it, because this is the kind of story you normally only hear about when you're 7 drinks in at midnight on Thursday at the Game Developers Conference.
As an aside, one of the books I recommend people read is Erik Bethke's Game Development and Production
. Bethke was one of the founders of Taldren, and his book actually includes some information on Black9 and lots of great information on how development process worked at Taldren.Labels: business, history, stories
Monday, April 21, 2008
Awesome News
So one of the few upcoming AAA titles I'm excited about is All Points Bulletin, from Realtime Worlds, the team that did Crackdown. Gamasutra says that they just bought the global distribution rights back from their publisher (presumably with the $50M in investment money they received recently). Crackdown was one of my favorite games of 2007, and APB looks amazing. I always love seeing great developers take back control of their work.
Labels: business
Thursday, April 17, 2008
EA, Take Two, Boston
So according to some, tonight we will know (more or less) the fate of EA's attempt at a hostile takeover of Take Two. This has me nervous: two of our biggest studios here in Boston are owned by Take Two (2K Boston, and, more recently, Rockstar New England). What could an EA buyout mean for the Boston area? Would they shut down 2K Boston? Not likely. What about RNE, which doesn't have a massive critical and sales hit game on their hands? As much as an EA takeover might mean good things for those studios, I'm looking at our development scene here and thinking that things are pretty good and I'd rather not see any major changes at all at this point.
Also of note is that Boston is pretty much the only major game development hub that doesn't have an EA-owned studio. (Okay, Seattle too, I think.)
There's some really good history and analysis of the buyout from Leigh Alexander (and a bunch of analysts) over at Kotaku.
Also of note is that Boston is pretty much the only major game development hub that doesn't have an EA-owned studio. (Okay, Seattle too, I think.)
There's some really good history and analysis of the buyout from Leigh Alexander (and a bunch of analysts) over at Kotaku.
